[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790520684602251-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 14:51:24 dist-test-slave-xk1g krb5kdc[399](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 14:51:24 dist-test-slave-xk1g krb5kdc[399](info): set up 2 sockets
Sep 27 14:51:24 dist-test-slave-xk1g krb5kdc[399](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260927 14:51:25.662449 392 mini_kdc.cc:121] Time spent starting KDC: real 1.024s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790520684602251-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790520684602251-392-0/krb5kdc/test-user.keytab.
Sep 27 14:51:25 dist-test-slave-xk1g krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790520685, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T14:51:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T14:51:25Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790520684602251-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790520684602251-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 14:51:25.784626 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6936m5/build/debug/bin/kudu
/tmp/dist-test-task6936m5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790520684602251-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790520684602251-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790520684602251-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790520684602251-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:32909
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790520684602251-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34053
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:32909
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790520684602251-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790520684602251-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790520684602251-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 14:51:25.894063 412 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 14:51:25.894323 412 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 14:51:25.894366 412 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 14:51:25.898355 412 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 14:51:25.898412 412 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 14:51:25.898432 412 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 14:51:25.898453 412 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 14:51:25.898469 412 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 14:51:25.903713 412 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34053
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790520684602251-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790520684602251-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:32909
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790520684602251-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:32909
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790520684602251-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.412
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790520684602251-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:15:17 UTC on ccycloud.quasar-ledxfe.root.comops.site
build id 4676134
I20260927 14:51:25.904709 412 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 14:51:25.905746 412 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 14:51:25.914235 420 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 14:51:25.914227 417 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 14:51:25.914506 412 server_base.cc:1034] running on GCE node
W20260927 14:51:25.914227 418 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 14:51:25.915163 412 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 14:51:25.916262 412 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 14:51:25.917446 412 hybrid_clock.cc:648] HybridClock initialized: now 1790520685917395 us; error 66 us; skew 500 ppm
Sep 27 14:51:25 dist-test-slave-xk1g krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790520685, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 14:51:25.920312 412 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 14:51:25.921263 412 webserver.cc:460] Webserver started at http://127.0.98.62:44879/ using document root <none> and password file <none>
I20260927 14:51:25.921826 412 fs_manager.cc:359] Metadata directory not provided
I20260927 14:51:25.921917 412 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 14:51:25.922150 412 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 14:51:25.924209 412 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790520684602251-392-0/minicluster-data/master-0/data/instance:
uuid: "7e1c72f4b3b34ae88e0c14f35521097e"
format_stamp: "Formatted at 2026-09-27 14:51:25 on dist-test-slave-xk1g"
server_key: "239e4aac972a0f5da3cddfc3646e1a61"
server_key_iv: "c67add6a419a0bfdd7450dbba2e0b83a"
server_key_version: "encryptionkey@0"
I20260927 14:51:25.924786 412 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790520684602251-392-0/minicluster-data/master-0/wal/instance:
uuid: "7e1c72f4b3b34ae88e0c14f35521097e"
format_stamp: "Formatted at 2026-09-27 14:51:25 on dist-test-slave-xk1g"
server_key: "239e4aac972a0f5da3cddfc3646e1a61"
server_key_iv: "c67add6a419a0bfdd7450dbba2e0b83a"
server_key_version: "encryptionkey@0"
I20260927 14:51:25.928680 412 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260927 14:51:25.931211 427 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 14:51:25.932497 412 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 14:51:25.932655 412 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790520684602251-392-0/minicluster-data/master-0/data,/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790520684602251-392-0/minicluster-data/master-0/wal
uuid: "7e1c72f4b3b34ae88e0c14f35521097e"
format_stamp: "Formatted at 2026-09-27 14:51:25 on dist-test-slave-xk1g"
server_key: "239e4aac972a0f5da3cddfc3646e1a61"
server_key_iv: "c67add6a419a0bfdd7450dbba2e0b83a"
server_key_version: "encryptionkey@0"
I20260927 14:51:25.932777 412 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790520684602251-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790520684602251-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790520684602251-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 14:51:25.947060 412 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 14:51:25.950456 412 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 14:51:25.950700 412 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 14:51:25.957618 412 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:32909
I20260927 14:51:25.957630 479 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:32909 every 8 connection(s)
I20260927 14:51:25.958556 412 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790520684602251-392-0/minicluster-data/master-0/data/info.pb
I20260927 14:51:25.960752 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6936m5/build/debug/bin/kudu as pid 412
I20260927 14:51:25.960912 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790520684602251-392-0/minicluster-data/master-0/wal/instance
I20260927 14:51:25.961376 480 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 14:51:25.962307 392 external_mini_cluster.cc:1428] Setting key 09b46086bd00257789e7f5e94e44304b
I20260927 14:51:25.966996 480 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7e1c72f4b3b34ae88e0c14f35521097e: Bootstrap starting.
I20260927 14:51:25.969168 480 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7e1c72f4b3b34ae88e0c14f35521097e: Neither blocks nor log segments found. Creating new log.
I20260927 14:51:25.970057 480 log.cc:824] T 00000000000000000000000000000000 P 7e1c72f4b3b34ae88e0c14f35521097e: Log is configured to *not* fsync() on all Append() calls
I20260927 14:51:25.972074 480 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7e1c72f4b3b34ae88e0c14f35521097e: No bootstrap required, opened a new log
I20260927 14:51:25.976235 480 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7e1c72f4b3b34ae88e0c14f35521097e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e1c72f4b3b34ae88e0c14f35521097e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32909 } }
I20260927 14:51:25.976497 480 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7e1c72f4b3b34ae88e0c14f35521097e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 14:51:25.976553 480 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7e1c72f4b3b34ae88e0c14f35521097e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7e1c72f4b3b34ae88e0c14f35521097e, State: Initialized, Role: FOLLOWER
I20260927 14:51:25.977012 480 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7e1c72f4b3b34ae88e0c14f35521097e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e1c72f4b3b34ae88e0c14f35521097e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32909 } }
I20260927 14:51:25.977205 480 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7e1c72f4b3b34ae88e0c14f35521097e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 14:51:25.977289 480 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7e1c72f4b3b34ae88e0c14f35521097e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 14:51:25.977385 480 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7e1c72f4b3b34ae88e0c14f35521097e [term 0 FOLLOWER]: Advancing to term 1
I20260927 14:51:25.978258 480 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7e1c72f4b3b34ae88e0c14f35521097e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e1c72f4b3b34ae88e0c14f35521097e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32909 } }
I20260927 14:51:25.978605 480 leader_election.cc:302] T 00000000000000000000000000000000 P 7e1c72f4b3b34ae88e0c14f35521097e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7e1c72f4b3b34ae88e0c14f35521097e; no voters:
I20260927 14:51:25.978950 480 leader_election.cc:288] T 00000000000000000000000000000000 P 7e1c72f4b3b34ae88e0c14f35521097e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 14:51:25.979004 485 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7e1c72f4b3b34ae88e0c14f35521097e [term 1 FOLLOWER]: Leader election won for term 1
I20260927 14:51:25.979185 485 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7e1c72f4b3b34ae88e0c14f35521097e [term 1 LEADER]: Becoming Leader. State: Replica: 7e1c72f4b3b34ae88e0c14f35521097e, State: Running, Role: LEADER
I20260927 14:51:25.979480 485 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7e1c72f4b3b34ae88e0c14f35521097e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e1c72f4b3b34ae88e0c14f35521097e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32909 } }
I20260927 14:51:25.980167 480 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7e1c72f4b3b34ae88e0c14f35521097e [sys.catalog]: configured and running, proceeding with master startup.
Sep 27 14:51:25 dist-test-slave-xk1g krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790520685, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 14:51:25.982779 486 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7e1c72f4b3b34ae88e0c14f35521097e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7e1c72f4b3b34ae88e0c14f35521097e. Latest consensus state: current_term: 1 leader_uuid: "7e1c72f4b3b34ae88e0c14f35521097e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e1c72f4b3b34ae88e0c14f35521097e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32909 } } }
I20260927 14:51:25.982987 487 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7e1c72f4b3b34ae88e0c14f35521097e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7e1c72f4b3b34ae88e0c14f35521097e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e1c72f4b3b34ae88e0c14f35521097e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32909 } } }
I20260927 14:51:25.983026 486 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7e1c72f4b3b34ae88e0c14f35521097e [sys.catalog]: This master's current role is: LEADER
I20260927 14:51:25.983052 487 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7e1c72f4b3b34ae88e0c14f35521097e [sys.catalog]: This master's current role is: LEADER
W20260927 14:51:25.985739 500 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 7e1c72f4b3b34ae88e0c14f35521097e: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 14:51:25.985827 500 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 14:51:25.985895 501 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 14:51:25.986562 501 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 14:51:25.988160 482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:51:25.965332 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51508 (local address 127.0.98.62:32909)
0927 14:51:25.965780 (+ 448us) server_negotiation.cc:206] Beginning negotiation
0927 14:51:25.965798 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0927 14:51:25.966863 (+ 1065us) server_negotiation.cc:407] Connection header received
0927 14:51:25.970097 (+ 3234us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:51:25.970118 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:51:25.970518 (+ 400us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:51:25.970789 (+ 271us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:51:25.973254 (+ 2465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:51:25.974783 (+ 1529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:51:25.976034 (+ 1251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:51:25.976698 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:51:25.976851 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:51:25.981848 (+ 4997us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:51:25.981855 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:51:25.981870 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 14:51:25.981905 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:51:25.984137 (+ 2232us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:51:25.984811 (+ 674us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:51:25.984817 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:51:25.984823 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:51:25.984967 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:51:25.985393 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:51:25.985397 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:51:25.985399 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:51:25.985916 (+ 517us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:51:25.986112 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0927 14:51:25.986937 (+ 825us) server_negotiation.cc:299] Negotiation successful
0927 14:51:25.987240 (+ 303us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":299,"thread_start_us":119,"threads_started":1}
I20260927 14:51:25.992728 501 catalog_manager.cc:1347] Generated new cluster ID: e4094c1a3e534fd6a74d614fde61a5a2
I20260927 14:51:25.992805 501 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 14:51:26.020961 501 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 14:51:26.022502 501 catalog_manager.cc:1504] Loading token signing keys...
I20260927 14:51:26.032424 501 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7e1c72f4b3b34ae88e0c14f35521097e: Generated new TSK 0
I20260927 14:51:26.033337 501 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790520684602251-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790520684602251-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260927 14:51:26.064147 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6936m5/build/debug/bin/kudu
/tmp/dist-test-task6936m5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790520684602251-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790520684602251-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790520684602251-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790520684602251-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:32909
--keytab_file=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790520684602251-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34053
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790520684602251-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790520684602251-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790520684602251-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 14:51:26.172865 506 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 14:51:26.173318 506 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 14:51:26.173388 506 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 14:51:26.177976 506 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 14:51:26.178031 506 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 14:51:26.178122 506 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 14:51:26.184305 506 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34053
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790520684602251-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790520684602251-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790520684602251-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790520684602251-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:32909
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.506
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790520684602251-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:15:17 UTC on ccycloud.quasar-ledxfe.root.comops.site
build id 4676134
I20260927 14:51:26.185446 506 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 14:51:26.186465 506 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 14:51:26.195912 512 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 14:51:26.196020 511 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 14:51:26.196106 506 server_base.cc:1034] running on GCE node
W20260927 14:51:26.195912 514 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 14:51:26.196710 506 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 14:51:26.197455 506 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 14:51:26.198652 506 hybrid_clock.cc:648] HybridClock initialized: now 1790520686198633 us; error 31 us; skew 500 ppm
Sep 27 14:51:26 dist-test-slave-xk1g krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790520686, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 14:51:26.201635 506 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 14:51:26.202735 506 webserver.cc:460] Webserver started at http://127.0.98.1:43799/ using document root <none> and password file <none>
I20260927 14:51:26.203298 506 fs_manager.cc:359] Metadata directory not provided
I20260927 14:51:26.203389 506 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 14:51:26.203619 506 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 14:51:26.205739 506 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790520684602251-392-0/minicluster-data/ts-0/data/instance:
uuid: "d0ec31776434442d864d1e05bf87d95f"
format_stamp: "Formatted at 2026-09-27 14:51:26 on dist-test-slave-xk1g"
server_key: "80edcc18887c15155a792c8ca80ea218"
server_key_iv: "0f24a313dcf5b4ab8f0781f4500b6fe4"
server_key_version: "encryptionkey@0"
I20260927 14:51:26.206285 506 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790520684602251-392-0/minicluster-data/ts-0/wal/instance:
uuid: "d0ec31776434442d864d1e05bf87d95f"
format_stamp: "Formatted at 2026-09-27 14:51:26 on dist-test-slave-xk1g"
server_key: "80edcc18887c15155a792c8ca80ea218"
server_key_iv: "0f24a313dcf5b4ab8f0781f4500b6fe4"
server_key_version: "encryptionkey@0"
I20260927 14:51:26.209868 506 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260927 14:51:26.212285 521 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 14:51:26.213599 506 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 14:51:26.213745 506 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790520684602251-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790520684602251-392-0/minicluster-data/ts-0/wal
uuid: "d0ec31776434442d864d1e05bf87d95f"
format_stamp: "Formatted at 2026-09-27 14:51:26 on dist-test-slave-xk1g"
server_key: "80edcc18887c15155a792c8ca80ea218"
server_key_iv: "0f24a313dcf5b4ab8f0781f4500b6fe4"
server_key_version: "encryptionkey@0"
I20260927 14:51:26.213860 506 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790520684602251-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790520684602251-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790520684602251-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 14:51:26.245620 506 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 14:51:26.248966 506 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 14:51:26.249195 506 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 14:51:26.249802 506 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 14:51:26.250622 506 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 14:51:26.250690 506 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 14:51:26.250761 506 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 14:51:26.250815 506 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 14:51:26.259634 506 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:37239
I20260927 14:51:26.259651 634 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:37239 every 8 connection(s)
I20260927 14:51:26.260550 506 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790520684602251-392-0/minicluster-data/ts-0/data/info.pb
Sep 27 14:51:26 dist-test-slave-xk1g krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790520686, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 14:51:26.270510 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6936m5/build/debug/bin/kudu as pid 506
I20260927 14:51:26.270597 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790520684602251-392-0/minicluster-data/ts-0/wal/instance
I20260927 14:51:26.270824 392 external_mini_cluster.cc:1428] Setting key aac7e632a2563f3f705306a682248832
I20260927 14:51:26.274605 482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:51:26.262556 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:44501 (local address 127.0.98.62:32909)
0927 14:51:26.262697 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0927 14:51:26.262701 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 14:51:26.263552 (+ 851us) server_negotiation.cc:407] Connection header received
0927 14:51:26.264916 (+ 1364us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:51:26.264919 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:51:26.264985 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:51:26.265112 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:51:26.266246 (+ 1134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:51:26.266773 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:51:26.267890 (+ 1117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:51:26.268131 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:51:26.268245 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:51:26.270866 (+ 2621us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:51:26.270870 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:51:26.270872 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 14:51:26.270895 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:51:26.272593 (+ 1698us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:51:26.273097 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:51:26.273102 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:51:26.273105 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:51:26.273184 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:51:26.273544 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:51:26.273549 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:51:26.273550 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:51:26.273721 (+ 171us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:51:26.273826 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0927 14:51:26.274310 (+ 484us) server_negotiation.cc:299] Negotiation successful
0927 14:51:26.274439 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260927 14:51:26.276046 637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:51:26.262923 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:32909 (local address 127.0.98.1:44501)
0927 14:51:26.263379 (+ 456us) negotiation.cc:107] Waiting for socket to connect
0927 14:51:26.263421 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0927 14:51:26.264700 (+ 1279us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:51:26.265284 (+ 584us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:51:26.265291 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:51:26.265650 (+ 359us) client_negotiation.cc:189] Negotiated authn=SASL
0927 14:51:26.266100 (+ 450us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:51:26.266112 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:51:26.266891 (+ 779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:51:26.266895 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:51:26.267765 (+ 870us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:51:26.267773 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:51:26.268280 (+ 507us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:51:26.268283 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:51:26.268448 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:51:26.269049 (+ 601us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 14:51:26.269118 (+ 69us) client_negotiation.cc:656] calling sasl_client_start()
0927 14:51:26.270685 (+ 1567us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 14:51:26.272709 (+ 2024us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 14:51:26.272714 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 14:51:26.272729 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0927 14:51:26.272981 (+ 252us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 14:51:26.273275 (+ 294us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 14:51:26.273280 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 14:51:26.273282 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 14:51:26.273463 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 14:51:26.273824 (+ 361us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 14:51:26.273831 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 14:51:26.274076 (+ 245us) client_negotiation.cc:769] Sending connection context
0927 14:51:26.274260 (+ 184us) client_negotiation.cc:240] Negotiation successful
0927 14:51:26.274577 (+ 317us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":261,"thread_start_us":100,"threads_started":1}
I20260927 14:51:26.277318 635 heartbeater.cc:347] Connected to a master server at 127.0.98.62:32909
I20260927 14:51:26.277558 635 heartbeater.cc:464] Registering TS with master...
I20260927 14:51:26.278033 635 heartbeater.cc:511] Master 127.0.98.62:32909 requested a full tablet report, sending...
I20260927 14:51:26.279544 444 ts_manager.cc:194] Registered new tserver with Master: d0ec31776434442d864d1e05bf87d95f (127.0.98.1:37239)
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260927 14:51:26.280853 444 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:44501
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790520684602251-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790520684602251-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260927 14:51:26.297482 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6936m5/build/debug/bin/kudu
/tmp/dist-test-task6936m5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790520684602251-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790520684602251-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790520684602251-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790520684602251-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:32909
--keytab_file=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790520684602251-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34053
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790520684602251-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790520684602251-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790520684602251-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 14:51:26.406790 640 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 14:51:26.407051 640 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 14:51:26.407095 640 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 14:51:26.410980 640 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 14:51:26.411036 640 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 14:51:26.411094 640 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 14:51:26.416272 640 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34053
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790520684602251-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790520684602251-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790520684602251-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790520684602251-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:32909
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.640
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790520684602251-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:15:17 UTC on ccycloud.quasar-ledxfe.root.comops.site
build id 4676134
I20260927 14:51:26.417339 640 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 14:51:26.418336 640 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 14:51:26.427775 648 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 14:51:26.427834 645 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 14:51:26.427776 646 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 14:51:26.428169 640 server_base.cc:1034] running on GCE node
I20260927 14:51:26.428576 640 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 14:51:26.429286 640 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 14:51:26.430480 640 hybrid_clock.cc:648] HybridClock initialized: now 1790520686430434 us; error 56 us; skew 500 ppm
Sep 27 14:51:26 dist-test-slave-xk1g krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790520686, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 14:51:26.433320 640 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 14:51:26.434433 640 webserver.cc:460] Webserver started at http://127.0.98.2:34181/ using document root <none> and password file <none>
I20260927 14:51:26.435153 640 fs_manager.cc:359] Metadata directory not provided
I20260927 14:51:26.435286 640 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 14:51:26.435559 640 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 14:51:26.438884 640 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790520684602251-392-0/minicluster-data/ts-1/data/instance:
uuid: "65328ea1ec2743e1b72e273221201c90"
format_stamp: "Formatted at 2026-09-27 14:51:26 on dist-test-slave-xk1g"
server_key: "a93aec32f7658726446039054145d47c"
server_key_iv: "08aa4bf60437d9b6ca75b1ecf0832c98"
server_key_version: "encryptionkey@0"
I20260927 14:51:26.439622 640 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790520684602251-392-0/minicluster-data/ts-1/wal/instance:
uuid: "65328ea1ec2743e1b72e273221201c90"
format_stamp: "Formatted at 2026-09-27 14:51:26 on dist-test-slave-xk1g"
server_key: "a93aec32f7658726446039054145d47c"
server_key_iv: "08aa4bf60437d9b6ca75b1ecf0832c98"
server_key_version: "encryptionkey@0"
I20260927 14:51:26.444134 640 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260927 14:51:26.446516 655 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 14:51:26.447722 640 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 14:51:26.447862 640 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790520684602251-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790520684602251-392-0/minicluster-data/ts-1/wal
uuid: "65328ea1ec2743e1b72e273221201c90"
format_stamp: "Formatted at 2026-09-27 14:51:26 on dist-test-slave-xk1g"
server_key: "a93aec32f7658726446039054145d47c"
server_key_iv: "08aa4bf60437d9b6ca75b1ecf0832c98"
server_key_version: "encryptionkey@0"
I20260927 14:51:26.447986 640 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790520684602251-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790520684602251-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790520684602251-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 14:51:26.466800 640 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 14:51:26.469718 640 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 14:51:26.469910 640 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 14:51:26.470469 640 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 14:51:26.471235 640 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 14:51:26.471282 640 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 14:51:26.471320 640 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 14:51:26.471370 640 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 14:51:26.481786 640 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:36053
I20260927 14:51:26.481827 768 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:36053 every 8 connection(s)
I20260927 14:51:26.482823 640 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790520684602251-392-0/minicluster-data/ts-1/data/info.pb
I20260927 14:51:26.483282 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6936m5/build/debug/bin/kudu as pid 640
I20260927 14:51:26.483394 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790520684602251-392-0/minicluster-data/ts-1/wal/instance
I20260927 14:51:26.483702 392 external_mini_cluster.cc:1428] Setting key 8310c618dd4fad0c6e4a132f6b6ffe56
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Sep 27 14:51:26 dist-test-slave-xk1g krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790520686, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 14:51:26.500643 482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:51:26.484527 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:39743 (local address 127.0.98.62:32909)
0927 14:51:26.484662 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0927 14:51:26.484666 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 14:51:26.485699 (+ 1033us) server_negotiation.cc:407] Connection header received
0927 14:51:26.487442 (+ 1743us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:51:26.487447 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:51:26.487503 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:51:26.487600 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:51:26.489090 (+ 1490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:51:26.490007 (+ 917us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:51:26.491111 (+ 1104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:51:26.491494 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:51:26.491604 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:51:26.496563 (+ 4959us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:51:26.496567 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:51:26.496576 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 14:51:26.496599 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:51:26.498378 (+ 1779us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:51:26.498889 (+ 511us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:51:26.498892 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:51:26.498894 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:51:26.498971 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:51:26.499334 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:51:26.499339 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:51:26.499341 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:51:26.499711 (+ 370us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:51:26.499847 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0927 14:51:26.500348 (+ 501us) server_negotiation.cc:299] Negotiation successful
0927 14:51:26.500480 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260927 14:51:26.501591 771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:51:26.485000 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:32909 (local address 127.0.98.2:39743)
0927 14:51:26.485529 (+ 529us) negotiation.cc:107] Waiting for socket to connect
0927 14:51:26.485587 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0927 14:51:26.487238 (+ 1651us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:51:26.487747 (+ 509us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:51:26.487753 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:51:26.488223 (+ 470us) client_negotiation.cc:189] Negotiated authn=SASL
0927 14:51:26.488888 (+ 665us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:51:26.488907 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:51:26.490128 (+ 1221us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:51:26.490130 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:51:26.490975 (+ 845us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:51:26.490992 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:51:26.491639 (+ 647us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:51:26.491641 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:51:26.491768 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:51:26.492362 (+ 594us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 14:51:26.492378 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 14:51:26.496362 (+ 3984us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 14:51:26.498501 (+ 2139us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 14:51:26.498506 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 14:51:26.498522 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0927 14:51:26.498775 (+ 253us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 14:51:26.499078 (+ 303us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 14:51:26.499081 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 14:51:26.499083 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 14:51:26.499236 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 14:51:26.499856 (+ 620us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 14:51:26.499864 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 14:51:26.500116 (+ 252us) client_negotiation.cc:769] Sending connection context
0927 14:51:26.500309 (+ 193us) client_negotiation.cc:240] Negotiation successful
0927 14:51:26.500657 (+ 348us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":309,"thread_start_us":94,"threads_started":1}
I20260927 14:51:26.502986 769 heartbeater.cc:347] Connected to a master server at 127.0.98.62:32909
I20260927 14:51:26.503223 769 heartbeater.cc:464] Registering TS with master...
I20260927 14:51:26.503746 769 heartbeater.cc:511] Master 127.0.98.62:32909 requested a full tablet report, sending...
I20260927 14:51:26.504735 444 ts_manager.cc:194] Registered new tserver with Master: 65328ea1ec2743e1b72e273221201c90 (127.0.98.2:36053)
I20260927 14:51:26.505352 444 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:39743
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790520684602251-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790520684602251-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260927 14:51:26.512197 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6936m5/build/debug/bin/kudu
/tmp/dist-test-task6936m5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790520684602251-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790520684602251-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790520684602251-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790520684602251-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:32909
--keytab_file=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790520684602251-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34053
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790520684602251-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790520684602251-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790520684602251-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 14:51:26.622447 774 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 14:51:26.622685 774 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 14:51:26.622753 774 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 14:51:26.626713 774 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 14:51:26.626801 774 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 14:51:26.626902 774 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 14:51:26.632279 774 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34053
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790520684602251-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790520684602251-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790520684602251-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790520684602251-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:32909
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.774
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790520684602251-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:15:17 UTC on ccycloud.quasar-ledxfe.root.comops.site
build id 4676134
I20260927 14:51:26.633293 774 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 14:51:26.634259 774 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 14:51:26.643381 782 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 14:51:26.643435 779 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 14:51:26.643455 780 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 14:51:26.643606 774 server_base.cc:1034] running on GCE node
I20260927 14:51:26.644227 774 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 14:51:26.644855 774 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 14:51:26.646063 774 hybrid_clock.cc:648] HybridClock initialized: now 1790520686646066 us; error 45 us; skew 500 ppm
Sep 27 14:51:26 dist-test-slave-xk1g krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790520686, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 14:51:26.648674 774 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 14:51:26.649518 774 webserver.cc:460] Webserver started at http://127.0.98.3:46599/ using document root <none> and password file <none>
I20260927 14:51:26.650009 774 fs_manager.cc:359] Metadata directory not provided
I20260927 14:51:26.650063 774 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 14:51:26.650210 774 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 14:51:26.652122 774 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790520684602251-392-0/minicluster-data/ts-2/data/instance:
uuid: "9e758ffe16c44087b84e11cf40a3b56f"
format_stamp: "Formatted at 2026-09-27 14:51:26 on dist-test-slave-xk1g"
server_key: "96cc87ea25a010752e3a8eb1029360f1"
server_key_iv: "4b140cfe144a67c85c7ad00834b56043"
server_key_version: "encryptionkey@0"
I20260927 14:51:26.652647 774 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790520684602251-392-0/minicluster-data/ts-2/wal/instance:
uuid: "9e758ffe16c44087b84e11cf40a3b56f"
format_stamp: "Formatted at 2026-09-27 14:51:26 on dist-test-slave-xk1g"
server_key: "96cc87ea25a010752e3a8eb1029360f1"
server_key_iv: "4b140cfe144a67c85c7ad00834b56043"
server_key_version: "encryptionkey@0"
I20260927 14:51:26.656112 774 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260927 14:51:26.658473 789 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 14:51:26.659777 774 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 14:51:26.659926 774 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790520684602251-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790520684602251-392-0/minicluster-data/ts-2/wal
uuid: "9e758ffe16c44087b84e11cf40a3b56f"
format_stamp: "Formatted at 2026-09-27 14:51:26 on dist-test-slave-xk1g"
server_key: "96cc87ea25a010752e3a8eb1029360f1"
server_key_iv: "4b140cfe144a67c85c7ad00834b56043"
server_key_version: "encryptionkey@0"
I20260927 14:51:26.660061 774 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790520684602251-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790520684602251-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790520684602251-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 14:51:26.698180 774 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 14:51:26.701174 774 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 14:51:26.701380 774 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 14:51:26.701984 774 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 14:51:26.702819 774 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 14:51:26.702867 774 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 14:51:26.702934 774 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 14:51:26.702979 774 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 14:51:26.712494 774 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:42863
I20260927 14:51:26.712523 902 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:42863 every 8 connection(s)
I20260927 14:51:26.713452 774 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790520684602251-392-0/minicluster-data/ts-2/data/info.pb
I20260927 14:51:26.718262 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6936m5/build/debug/bin/kudu as pid 774
I20260927 14:51:26.718495 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790520684602251-392-0/minicluster-data/ts-2/wal/instance
I20260927 14:51:26.718806 392 external_mini_cluster.cc:1428] Setting key bce6adc00f8a3a5f0410a49b28b94adb
Sep 27 14:51:26 dist-test-slave-xk1g krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790520686, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 14:51:26.727273 482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:51:26.715240 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:38909 (local address 127.0.98.62:32909)
0927 14:51:26.715374 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0927 14:51:26.715378 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 14:51:26.716184 (+ 806us) server_negotiation.cc:407] Connection header received
0927 14:51:26.717523 (+ 1339us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:51:26.717527 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:51:26.717579 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:51:26.717687 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:51:26.718943 (+ 1256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:51:26.719524 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:51:26.721124 (+ 1600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:51:26.721386 (+ 262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:51:26.721511 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:51:26.723943 (+ 2432us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:51:26.723946 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:51:26.723948 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 14:51:26.723971 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:51:26.725293 (+ 1322us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:51:26.725762 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:51:26.725765 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:51:26.725766 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:51:26.725849 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:51:26.726230 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:51:26.726232 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:51:26.726233 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:51:26.726411 (+ 178us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:51:26.726505 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0927 14:51:26.727046 (+ 541us) server_negotiation.cc:299] Negotiation successful
0927 14:51:26.727150 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260927 14:51:26.728101 905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:51:26.715616 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:32909 (local address 127.0.98.3:38909)
0927 14:51:26.716038 (+ 422us) negotiation.cc:107] Waiting for socket to connect
0927 14:51:26.716079 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0927 14:51:26.717316 (+ 1237us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:51:26.717852 (+ 536us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:51:26.717859 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:51:26.718216 (+ 357us) client_negotiation.cc:189] Negotiated authn=SASL
0927 14:51:26.718775 (+ 559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:51:26.718788 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:51:26.719667 (+ 879us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:51:26.719670 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:51:26.720934 (+ 1264us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:51:26.720945 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:51:26.721557 (+ 612us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:51:26.721560 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:51:26.721705 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:51:26.722397 (+ 692us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 14:51:26.722412 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 14:51:26.723797 (+ 1385us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 14:51:26.725394 (+ 1597us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 14:51:26.725399 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 14:51:26.725411 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0927 14:51:26.725655 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 14:51:26.725975 (+ 320us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 14:51:26.725978 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 14:51:26.725979 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 14:51:26.726140 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 14:51:26.726538 (+ 398us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 14:51:26.726543 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 14:51:26.726767 (+ 224us) client_negotiation.cc:769] Sending connection context
0927 14:51:26.726934 (+ 167us) client_negotiation.cc:240] Negotiation successful
0927 14:51:26.727194 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":241,"thread_start_us":103,"threads_started":1}
I20260927 14:51:26.729341 903 heartbeater.cc:347] Connected to a master server at 127.0.98.62:32909
I20260927 14:51:26.729565 903 heartbeater.cc:464] Registering TS with master...
I20260927 14:51:26.730000 903 heartbeater.cc:511] Master 127.0.98.62:32909 requested a full tablet report, sending...
I20260927 14:51:26.731004 444 ts_manager.cc:194] Registered new tserver with Master: 9e758ffe16c44087b84e11cf40a3b56f (127.0.98.3:42863)
I20260927 14:51:26.731592 444 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:38909
I20260927 14:51:26.733403 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 27 14:51:26 dist-test-slave-xk1g krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790520686, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 27 14:51:26 dist-test-slave-xk1g krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790520686, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260927 14:51:26.759594 910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:51:26.750121 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42152 (local address 127.0.98.1:37239)
0927 14:51:26.750433 (+ 312us) server_negotiation.cc:206] Beginning negotiation
0927 14:51:26.750445 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 14:51:26.750472 (+ 27us) server_negotiation.cc:407] Connection header received
0927 14:51:26.750574 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:51:26.750580 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:51:26.750764 (+ 184us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:51:26.750928 (+ 164us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:51:26.751561 (+ 633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:51:26.752391 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:51:26.753342 (+ 951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:51:26.753727 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:51:26.753847 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:51:26.756128 (+ 2281us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:51:26.756136 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:51:26.756150 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 14:51:26.756172 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:51:26.757908 (+ 1736us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:51:26.758293 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:51:26.758298 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:51:26.758299 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:51:26.758370 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:51:26.758632 (+ 262us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:51:26.758635 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:51:26.758637 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:51:26.758872 (+ 235us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:51:26.759031 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0927 14:51:26.759211 (+ 180us) server_negotiation.cc:299] Negotiation successful
0927 14:51:26.759392 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":218,"thread_start_us":106,"threads_started":1}
W20260927 14:51:26.760552 549 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42152
Sep 27 14:51:26 dist-test-slave-xk1g krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790520686, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 27 14:51:26 dist-test-slave-xk1g krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790520686, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260927 14:51:26.785964 910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:51:26.777853 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42156 (local address 127.0.98.1:37239)
0927 14:51:26.778028 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0927 14:51:26.778032 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 14:51:26.778049 (+ 17us) server_negotiation.cc:407] Connection header received
0927 14:51:26.778185 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:51:26.778187 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:51:26.778227 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:51:26.778299 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:51:26.778929 (+ 630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:51:26.779448 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:51:26.780445 (+ 997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:51:26.780720 (+ 275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:51:26.780801 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:51:26.783187 (+ 2386us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:51:26.783190 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:51:26.783192 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 14:51:26.783214 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:51:26.784479 (+ 1265us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:51:26.784829 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:51:26.784832 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:51:26.784833 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:51:26.784919 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:51:26.785209 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:51:26.785211 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:51:26.785212 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:51:26.785359 (+ 147us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:51:26.785454 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0927 14:51:26.785658 (+ 204us) server_negotiation.cc:299] Negotiation successful
0927 14:51:26.785749 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260927 14:51:26.786899 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6936m5/build/debug/bin/kudu with pid 506
I20260927 14:51:26.794168 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6936m5/build/debug/bin/kudu with pid 640
I20260927 14:51:26.801510 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6936m5/build/debug/bin/kudu with pid 774
I20260927 14:51:26.806505 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6936m5/build/debug/bin/kudu with pid 412
2026-09-27T14:51:26Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (2198 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790520684602251-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 14:51:26 dist-test-slave-xk1g krb5kdc[917](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 14:51:26 dist-test-slave-xk1g krb5kdc[917](info): set up 2 sockets
Sep 27 14:51:26 dist-test-slave-xk1g krb5kdc[917](info): commencing operation
krb5kdc: starting...
W20260927 14:51:27.895617 392 mini_kdc.cc:121] Time spent starting KDC: real 1.067s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
W20260927 14:51:28.088754 392 mini_kdc.cc:253] Time spent creating user principal test-admin: real 0.193s user 0.001s sys 0.001s
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790520684602251-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790520684602251-392-0/krb5kdc/test-user.keytab.
Sep 27 14:51:28 dist-test-slave-xk1g krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790520688, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T14:51:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T14:51:28Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790520684602251-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790520684602251-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 14:51:28.365177 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6936m5/build/debug/bin/kudu
/tmp/dist-test-task6936m5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790520684602251-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790520684602251-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790520684602251-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790520684602251-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34249
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790520684602251-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35367
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:34249
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790520684602251-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790520684602251-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790520684602251-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 14:51:28.472322 930 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 14:51:28.472579 930 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 14:51:28.472618 930 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 14:51:28.476428 930 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 14:51:28.476481 930 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 14:51:28.476502 930 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 14:51:28.476518 930 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 14:51:28.476533 930 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 14:51:28.481654 930 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35367
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790520684602251-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790520684602251-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:34249
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790520684602251-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34249
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790520684602251-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.930
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790520684602251-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:15:17 UTC on ccycloud.quasar-ledxfe.root.comops.site
build id 4676134
I20260927 14:51:28.482556 930 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 14:51:28.483460 930 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 14:51:28.491833 938 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 14:51:28.491827 935 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 14:51:28.491926 936 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 14:51:28.492053 930 server_base.cc:1034] running on GCE node
I20260927 14:51:28.492671 930 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 14:51:28.493741 930 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 14:51:28.494933 930 hybrid_clock.cc:648] HybridClock initialized: now 1790520688494926 us; error 29 us; skew 500 ppm
Sep 27 14:51:28 dist-test-slave-xk1g krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790520688, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 14:51:28.497566 930 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 14:51:28.498396 930 webserver.cc:460] Webserver started at http://127.0.98.62:41801/ using document root <none> and password file <none>
I20260927 14:51:28.498914 930 fs_manager.cc:359] Metadata directory not provided
I20260927 14:51:28.498996 930 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 14:51:28.499223 930 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 14:51:28.501199 930 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790520684602251-392-0/minicluster-data/master-0/data/instance:
uuid: "7d91750a4f7e4fab92336615eeb35343"
format_stamp: "Formatted at 2026-09-27 14:51:28 on dist-test-slave-xk1g"
server_key: "8b41380f4f939eefdcc5c66dee84dbad"
server_key_iv: "0db1f5235fde630130daad6c84bf4afc"
server_key_version: "encryptionkey@0"
I20260927 14:51:28.501967 930 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790520684602251-392-0/minicluster-data/master-0/wal/instance:
uuid: "7d91750a4f7e4fab92336615eeb35343"
format_stamp: "Formatted at 2026-09-27 14:51:28 on dist-test-slave-xk1g"
server_key: "8b41380f4f939eefdcc5c66dee84dbad"
server_key_iv: "0db1f5235fde630130daad6c84bf4afc"
server_key_version: "encryptionkey@0"
I20260927 14:51:28.505964 930 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.002s
I20260927 14:51:28.508268 945 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 14:51:28.509454 930 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 14:51:28.509580 930 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790520684602251-392-0/minicluster-data/master-0/data,/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790520684602251-392-0/minicluster-data/master-0/wal
uuid: "7d91750a4f7e4fab92336615eeb35343"
format_stamp: "Formatted at 2026-09-27 14:51:28 on dist-test-slave-xk1g"
server_key: "8b41380f4f939eefdcc5c66dee84dbad"
server_key_iv: "0db1f5235fde630130daad6c84bf4afc"
server_key_version: "encryptionkey@0"
I20260927 14:51:28.509689 930 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790520684602251-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790520684602251-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790520684602251-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 14:51:28.525588 930 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 14:51:28.528326 930 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 14:51:28.528524 930 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 14:51:28.535440 930 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:34249
I20260927 14:51:28.535495 997 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:34249 every 8 connection(s)
I20260927 14:51:28.536386 930 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790520684602251-392-0/minicluster-data/master-0/data/info.pb
I20260927 14:51:28.539098 998 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 14:51:28.540766 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6936m5/build/debug/bin/kudu as pid 930
I20260927 14:51:28.540887 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790520684602251-392-0/minicluster-data/master-0/wal/instance
I20260927 14:51:28.541193 392 external_mini_cluster.cc:1428] Setting key a16b122565b9b4c5f6efec47c4aef187
I20260927 14:51:28.545686 998 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7d91750a4f7e4fab92336615eeb35343: Bootstrap starting.
I20260927 14:51:28.548274 998 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7d91750a4f7e4fab92336615eeb35343: Neither blocks nor log segments found. Creating new log.
I20260927 14:51:28.549187 998 log.cc:824] T 00000000000000000000000000000000 P 7d91750a4f7e4fab92336615eeb35343: Log is configured to *not* fsync() on all Append() calls
Sep 27 14:51:28 dist-test-slave-xk1g krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790520688, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 14:51:28.551396 998 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7d91750a4f7e4fab92336615eeb35343: No bootstrap required, opened a new log
I20260927 14:51:28.555347 998 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7d91750a4f7e4fab92336615eeb35343 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d91750a4f7e4fab92336615eeb35343" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34249 } }
I20260927 14:51:28.555598 998 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7d91750a4f7e4fab92336615eeb35343 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 14:51:28.555671 998 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7d91750a4f7e4fab92336615eeb35343 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7d91750a4f7e4fab92336615eeb35343, State: Initialized, Role: FOLLOWER
I20260927 14:51:28.555859 1001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:51:28.542489 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48818 (local address 127.0.98.62:34249)
0927 14:51:28.542889 (+ 400us) server_negotiation.cc:206] Beginning negotiation
0927 14:51:28.542908 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0927 14:51:28.542948 (+ 40us) server_negotiation.cc:407] Connection header received
0927 14:51:28.543799 (+ 851us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:51:28.543824 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:51:28.544230 (+ 406us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:51:28.544603 (+ 373us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:51:28.545250 (+ 647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:51:28.546652 (+ 1402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:51:28.547457 (+ 805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:51:28.548124 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:51:28.548300 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:51:28.551172 (+ 2872us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:51:28.551177 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:51:28.551190 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 14:51:28.551221 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:51:28.552913 (+ 1692us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:51:28.553521 (+ 608us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:51:28.553526 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:51:28.553530 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:51:28.553633 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:51:28.553913 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:51:28.553915 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:51:28.553916 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:51:28.554305 (+ 389us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:51:28.554461 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0927 14:51:28.554739 (+ 278us) server_negotiation.cc:299] Negotiation successful
0927 14:51:28.555061 (+ 322us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":262,"thread_start_us":108,"threads_started":1}
I20260927 14:51:28.556164 998 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7d91750a4f7e4fab92336615eeb35343 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d91750a4f7e4fab92336615eeb35343" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34249 } }
I20260927 14:51:28.556319 998 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7d91750a4f7e4fab92336615eeb35343 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 14:51:28.556398 998 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7d91750a4f7e4fab92336615eeb35343 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 14:51:28.556500 998 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7d91750a4f7e4fab92336615eeb35343 [term 0 FOLLOWER]: Advancing to term 1
I20260927 14:51:28.557597 998 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7d91750a4f7e4fab92336615eeb35343 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d91750a4f7e4fab92336615eeb35343" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34249 } }
I20260927 14:51:28.557938 998 leader_election.cc:302] T 00000000000000000000000000000000 P 7d91750a4f7e4fab92336615eeb35343 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7d91750a4f7e4fab92336615eeb35343; no voters:
I20260927 14:51:28.558287 998 leader_election.cc:288] T 00000000000000000000000000000000 P 7d91750a4f7e4fab92336615eeb35343 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 14:51:28.558423 1003 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7d91750a4f7e4fab92336615eeb35343 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 14:51:28.558655 1003 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7d91750a4f7e4fab92336615eeb35343 [term 1 LEADER]: Becoming Leader. State: Replica: 7d91750a4f7e4fab92336615eeb35343, State: Running, Role: LEADER
I20260927 14:51:28.558962 1003 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7d91750a4f7e4fab92336615eeb35343 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d91750a4f7e4fab92336615eeb35343" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34249 } }
I20260927 14:51:28.559480 998 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7d91750a4f7e4fab92336615eeb35343 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 14:51:28.560619 1004 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7d91750a4f7e4fab92336615eeb35343 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7d91750a4f7e4fab92336615eeb35343" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d91750a4f7e4fab92336615eeb35343" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34249 } } }
I20260927 14:51:28.560755 1004 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7d91750a4f7e4fab92336615eeb35343 [sys.catalog]: This master's current role is: LEADER
I20260927 14:51:28.561028 1005 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7d91750a4f7e4fab92336615eeb35343 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7d91750a4f7e4fab92336615eeb35343. Latest consensus state: current_term: 1 leader_uuid: "7d91750a4f7e4fab92336615eeb35343" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d91750a4f7e4fab92336615eeb35343" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34249 } } }
I20260927 14:51:28.561175 1005 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7d91750a4f7e4fab92336615eeb35343 [sys.catalog]: This master's current role is: LEADER
I20260927 14:51:28.561141 1012 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 14:51:28.564473 1012 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 14:51:28.570127 1012 catalog_manager.cc:1347] Generated new cluster ID: f4422c447d554bd6b2cba9ebad44c9ea
I20260927 14:51:28.570202 1012 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 14:51:28.580505 1012 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 14:51:28.581682 1012 catalog_manager.cc:1504] Loading token signing keys...
I20260927 14:51:28.585809 1012 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7d91750a4f7e4fab92336615eeb35343: Generated new TSK 0
I20260927 14:51:28.586422 1012 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790520684602251-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790520684602251-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260927 14:51:28.617885 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6936m5/build/debug/bin/kudu
/tmp/dist-test-task6936m5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790520684602251-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790520684602251-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790520684602251-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790520684602251-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34249
--keytab_file=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790520684602251-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35367
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790520684602251-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790520684602251-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790520684602251-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 14:51:28.748924 1024 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 14:51:28.749217 1024 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 14:51:28.749261 1024 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 14:51:28.753197 1024 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 14:51:28.753252 1024 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 14:51:28.753309 1024 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 14:51:28.758505 1024 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35367
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790520684602251-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790520684602251-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790520684602251-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790520684602251-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34249
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1024
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790520684602251-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:15:17 UTC on ccycloud.quasar-ledxfe.root.comops.site
build id 4676134
I20260927 14:51:28.759469 1024 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 14:51:28.760486 1024 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 14:51:28.770318 1029 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 14:51:28.770320 1030 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 14:51:28.770691 1024 server_base.cc:1034] running on GCE node
W20260927 14:51:28.770800 1032 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 14:51:28.771201 1024 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 14:51:28.771832 1024 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 14:51:28.773008 1024 hybrid_clock.cc:648] HybridClock initialized: now 1790520688772991 us; error 28 us; skew 500 ppm
Sep 27 14:51:28 dist-test-slave-xk1g krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790520688, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 14:51:28.776062 1024 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 14:51:28.776975 1024 webserver.cc:460] Webserver started at http://127.0.98.1:36985/ using document root <none> and password file <none>
I20260927 14:51:28.777554 1024 fs_manager.cc:359] Metadata directory not provided
I20260927 14:51:28.777649 1024 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 14:51:28.777868 1024 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 14:51:28.779923 1024 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790520684602251-392-0/minicluster-data/ts-0/data/instance:
uuid: "ebe94da1eee14c91b9f5177e82ea5090"
format_stamp: "Formatted at 2026-09-27 14:51:28 on dist-test-slave-xk1g"
server_key: "47f0b59961d83cc20b1f4a82a2fba380"
server_key_iv: "67d36067daf5586960493d13ef7d921c"
server_key_version: "encryptionkey@0"
I20260927 14:51:28.780465 1024 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790520684602251-392-0/minicluster-data/ts-0/wal/instance:
uuid: "ebe94da1eee14c91b9f5177e82ea5090"
format_stamp: "Formatted at 2026-09-27 14:51:28 on dist-test-slave-xk1g"
server_key: "47f0b59961d83cc20b1f4a82a2fba380"
server_key_iv: "67d36067daf5586960493d13ef7d921c"
server_key_version: "encryptionkey@0"
I20260927 14:51:28.783979 1024 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260927 14:51:28.786329 1039 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 14:51:28.787442 1024 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 14:51:28.787566 1024 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790520684602251-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790520684602251-392-0/minicluster-data/ts-0/wal
uuid: "ebe94da1eee14c91b9f5177e82ea5090"
format_stamp: "Formatted at 2026-09-27 14:51:28 on dist-test-slave-xk1g"
server_key: "47f0b59961d83cc20b1f4a82a2fba380"
server_key_iv: "67d36067daf5586960493d13ef7d921c"
server_key_version: "encryptionkey@0"
I20260927 14:51:28.787676 1024 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790520684602251-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790520684602251-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790520684602251-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 14:51:28.803437 1024 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 14:51:28.806697 1024 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 14:51:28.806892 1024 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 14:51:28.807459 1024 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 14:51:28.808260 1024 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 14:51:28.808324 1024 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 14:51:28.808395 1024 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 14:51:28.808434 1024 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 14:51:28.817425 1024 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:33751
I20260927 14:51:28.817444 1152 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:33751 every 8 connection(s)
I20260927 14:51:28.818333 1024 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790520684602251-392-0/minicluster-data/ts-0/data/info.pb
I20260927 14:51:28.824776 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6936m5/build/debug/bin/kudu as pid 1024
I20260927 14:51:28.824859 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790520684602251-392-0/minicluster-data/ts-0/wal/instance
I20260927 14:51:28.825196 392 external_mini_cluster.cc:1428] Setting key 6dda9fb34bf216e8213560a888d189aa
Sep 27 14:51:28 dist-test-slave-xk1g krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790520688, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 14:51:28.833109 1001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:51:28.820101 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:56563 (local address 127.0.98.62:34249)
0927 14:51:28.820246 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0927 14:51:28.820250 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 14:51:28.821088 (+ 838us) server_negotiation.cc:407] Connection header received
0927 14:51:28.822520 (+ 1432us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:51:28.822522 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:51:28.822568 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:51:28.822648 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:51:28.823832 (+ 1184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:51:28.824377 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:51:28.825777 (+ 1400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:51:28.826112 (+ 335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:51:28.826207 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:51:28.829235 (+ 3028us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:51:28.829238 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:51:28.829240 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 14:51:28.829264 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:51:28.830724 (+ 1460us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:51:28.831401 (+ 677us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:51:28.831403 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:51:28.831405 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:51:28.831474 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:51:28.831891 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:51:28.831893 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:51:28.831895 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:51:28.832065 (+ 170us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:51:28.832134 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0927 14:51:28.832831 (+ 697us) server_negotiation.cc:299] Negotiation successful
0927 14:51:28.832934 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260927 14:51:28.834375 1155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:51:28.820449 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34249 (local address 127.0.98.1:56563)
0927 14:51:28.820921 (+ 472us) negotiation.cc:107] Waiting for socket to connect
0927 14:51:28.820965 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0927 14:51:28.822288 (+ 1323us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:51:28.822807 (+ 519us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:51:28.822814 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:51:28.823205 (+ 391us) client_negotiation.cc:189] Negotiated authn=SASL
0927 14:51:28.823689 (+ 484us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:51:28.823703 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:51:28.824490 (+ 787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:51:28.824493 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:51:28.825622 (+ 1129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:51:28.825632 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:51:28.826267 (+ 635us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:51:28.826270 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:51:28.826504 (+ 234us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:51:28.827131 (+ 627us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 14:51:28.827148 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 14:51:28.829022 (+ 1874us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 14:51:28.830843 (+ 1821us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 14:51:28.830854 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 14:51:28.830874 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0927 14:51:28.831287 (+ 413us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 14:51:28.831566 (+ 279us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 14:51:28.831571 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 14:51:28.831577 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0927 14:51:28.831788 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 14:51:28.832171 (+ 383us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 14:51:28.832182 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 14:51:28.832560 (+ 378us) client_negotiation.cc:769] Sending connection context
0927 14:51:28.832797 (+ 237us) client_negotiation.cc:240] Negotiation successful
0927 14:51:28.833203 (+ 406us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":275,"thread_start_us":131,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260927 14:51:28.835841 1153 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34249
I20260927 14:51:28.836102 1153 heartbeater.cc:464] Registering TS with master...
I20260927 14:51:28.836617 1153 heartbeater.cc:511] Master 127.0.98.62:34249 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260927 14:51:28.838415 962 ts_manager.cc:194] Registered new tserver with Master: ebe94da1eee14c91b9f5177e82ea5090 (127.0.98.1:33751)
I20260927 14:51:28.839533 962 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:56563
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790520684602251-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790520684602251-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260927 14:51:28.854614 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6936m5/build/debug/bin/kudu
/tmp/dist-test-task6936m5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790520684602251-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790520684602251-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790520684602251-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790520684602251-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34249
--keytab_file=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790520684602251-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35367
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790520684602251-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790520684602251-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790520684602251-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 14:51:28.965241 1158 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 14:51:28.965553 1158 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 14:51:28.965601 1158 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 14:51:28.970152 1158 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 14:51:28.970221 1158 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 14:51:28.970285 1158 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 14:51:28.975800 1158 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35367
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790520684602251-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790520684602251-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790520684602251-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790520684602251-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34249
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1158
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790520684602251-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:15:17 UTC on ccycloud.quasar-ledxfe.root.comops.site
build id 4676134
I20260927 14:51:28.976840 1158 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 14:51:28.977921 1158 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 14:51:28.987691 1163 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 14:51:28.987838 1158 server_base.cc:1034] running on GCE node
W20260927 14:51:28.987723 1166 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 14:51:28.987704 1164 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 14:51:28.988408 1158 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 14:51:28.989166 1158 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 14:51:28.990351 1158 hybrid_clock.cc:648] HybridClock initialized: now 1790520688990329 us; error 44 us; skew 500 ppm
Sep 27 14:51:28 dist-test-slave-xk1g krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790520688, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 14:51:28.993364 1158 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 14:51:28.994259 1158 webserver.cc:460] Webserver started at http://127.0.98.2:43747/ using document root <none> and password file <none>
I20260927 14:51:28.994844 1158 fs_manager.cc:359] Metadata directory not provided
I20260927 14:51:28.994932 1158 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 14:51:28.995163 1158 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 14:51:28.997380 1158 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790520684602251-392-0/minicluster-data/ts-1/data/instance:
uuid: "2bf38eb63cfc4236b6a6fd3261bad5c3"
format_stamp: "Formatted at 2026-09-27 14:51:28 on dist-test-slave-xk1g"
server_key: "1436057ae260f4da44155486898e7fa1"
server_key_iv: "902f020f4c98635ab5cc2875d921410f"
server_key_version: "encryptionkey@0"
I20260927 14:51:28.997974 1158 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790520684602251-392-0/minicluster-data/ts-1/wal/instance:
uuid: "2bf38eb63cfc4236b6a6fd3261bad5c3"
format_stamp: "Formatted at 2026-09-27 14:51:28 on dist-test-slave-xk1g"
server_key: "1436057ae260f4da44155486898e7fa1"
server_key_iv: "902f020f4c98635ab5cc2875d921410f"
server_key_version: "encryptionkey@0"
I20260927 14:51:29.002030 1158 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260927 14:51:29.004527 1173 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 14:51:29.005889 1158 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 14:51:29.006038 1158 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790520684602251-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790520684602251-392-0/minicluster-data/ts-1/wal
uuid: "2bf38eb63cfc4236b6a6fd3261bad5c3"
format_stamp: "Formatted at 2026-09-27 14:51:28 on dist-test-slave-xk1g"
server_key: "1436057ae260f4da44155486898e7fa1"
server_key_iv: "902f020f4c98635ab5cc2875d921410f"
server_key_version: "encryptionkey@0"
I20260927 14:51:29.006276 1158 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790520684602251-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790520684602251-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790520684602251-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 14:51:29.037643 1158 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 14:51:29.040846 1158 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 14:51:29.041091 1158 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 14:51:29.041707 1158 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 14:51:29.042569 1158 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 14:51:29.042635 1158 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 14:51:29.042704 1158 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 14:51:29.042743 1158 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 14:51:29.051447 1158 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:43027
I20260927 14:51:29.051486 1286 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:43027 every 8 connection(s)
I20260927 14:51:29.052340 1158 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790520684602251-392-0/minicluster-data/ts-1/data/info.pb
I20260927 14:51:29.061223 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6936m5/build/debug/bin/kudu as pid 1158
I20260927 14:51:29.061321 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790520684602251-392-0/minicluster-data/ts-1/wal/instance
I20260927 14:51:29.061712 392 external_mini_cluster.cc:1428] Setting key 3e1c2f50c84adef06e3f7eaca3a4558b
Sep 27 14:51:29 dist-test-slave-xk1g krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790520688, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 14:51:29.066607 1001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:51:29.054230 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:34803 (local address 127.0.98.62:34249)
0927 14:51:29.054373 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0927 14:51:29.054378 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 14:51:29.055371 (+ 993us) server_negotiation.cc:407] Connection header received
0927 14:51:29.056676 (+ 1305us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:51:29.056679 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:51:29.056724 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:51:29.056803 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:51:29.058039 (+ 1236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:51:29.058585 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:51:29.059754 (+ 1169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:51:29.059972 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:51:29.060077 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:51:29.062861 (+ 2784us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:51:29.062865 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:51:29.062868 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 14:51:29.062898 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:51:29.064453 (+ 1555us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:51:29.065011 (+ 558us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:51:29.065014 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:51:29.065020 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:51:29.065154 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:51:29.065502 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:51:29.065506 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:51:29.065508 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:51:29.065691 (+ 183us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:51:29.065788 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0927 14:51:29.066306 (+ 518us) server_negotiation.cc:299] Negotiation successful
0927 14:51:29.066436 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260927 14:51:29.067638 1289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:51:29.054670 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34249 (local address 127.0.98.2:34803)
0927 14:51:29.055179 (+ 509us) negotiation.cc:107] Waiting for socket to connect
0927 14:51:29.055244 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0927 14:51:29.056465 (+ 1221us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:51:29.056960 (+ 495us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:51:29.056966 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:51:29.057380 (+ 414us) client_negotiation.cc:189] Negotiated authn=SASL
0927 14:51:29.057865 (+ 485us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:51:29.057878 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:51:29.058701 (+ 823us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:51:29.058704 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:51:29.059645 (+ 941us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:51:29.059654 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:51:29.060110 (+ 456us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:51:29.060112 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:51:29.060247 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:51:29.060885 (+ 638us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 14:51:29.060900 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 14:51:29.062700 (+ 1800us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 14:51:29.064587 (+ 1887us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 14:51:29.064594 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 14:51:29.064610 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0927 14:51:29.064883 (+ 273us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 14:51:29.065247 (+ 364us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 14:51:29.065250 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 14:51:29.065252 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 14:51:29.065403 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 14:51:29.065802 (+ 399us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 14:51:29.065811 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 14:51:29.066071 (+ 260us) client_negotiation.cc:769] Sending connection context
0927 14:51:29.066265 (+ 194us) client_negotiation.cc:240] Negotiation successful
0927 14:51:29.066606 (+ 341us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":248,"thread_start_us":116,"threads_started":1}
I20260927 14:51:29.068957 1287 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34249
I20260927 14:51:29.069255 1287 heartbeater.cc:464] Registering TS with master...
I20260927 14:51:29.069779 1287 heartbeater.cc:511] Master 127.0.98.62:34249 requested a full tablet report, sending...
I20260927 14:51:29.070904 962 ts_manager.cc:194] Registered new tserver with Master: 2bf38eb63cfc4236b6a6fd3261bad5c3 (127.0.98.2:43027)
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260927 14:51:29.071523 962 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:34803
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790520684602251-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790520684602251-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260927 14:51:29.088558 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6936m5/build/debug/bin/kudu
/tmp/dist-test-task6936m5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790520684602251-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790520684602251-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790520684602251-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790520684602251-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34249
--keytab_file=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790520684602251-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35367
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790520684602251-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790520684602251-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790520684602251-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 14:51:29.197782 1292 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 14:51:29.198041 1292 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 14:51:29.198109 1292 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 14:51:29.202155 1292 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 14:51:29.202211 1292 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 14:51:29.202298 1292 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 14:51:29.207473 1292 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35367
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790520684602251-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790520684602251-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790520684602251-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790520684602251-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34249
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1292
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790520684602251-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:15:17 UTC on ccycloud.quasar-ledxfe.root.comops.site
build id 4676134
I20260927 14:51:29.208523 1292 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 14:51:29.209538 1292 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 14:51:29.218971 1298 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 14:51:29.219041 1297 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 14:51:29.218994 1300 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 14:51:29.219115 1292 server_base.cc:1034] running on GCE node
I20260927 14:51:29.219522 1292 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 14:51:29.220180 1292 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 14:51:29.221366 1292 hybrid_clock.cc:648] HybridClock initialized: now 1790520689221341 us; error 37 us; skew 500 ppm
Sep 27 14:51:29 dist-test-slave-xk1g krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790520689, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 14:51:29.224148 1292 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 14:51:29.224946 1292 webserver.cc:460] Webserver started at http://127.0.98.3:43597/ using document root <none> and password file <none>
I20260927 14:51:29.225484 1292 fs_manager.cc:359] Metadata directory not provided
I20260927 14:51:29.225543 1292 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 14:51:29.225740 1292 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 14:51:29.227674 1292 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790520684602251-392-0/minicluster-data/ts-2/data/instance:
uuid: "dcfd6b8afc3a459fa181d304fcf6054b"
format_stamp: "Formatted at 2026-09-27 14:51:29 on dist-test-slave-xk1g"
server_key: "32f94ed253c1caaaa1c41c74aacf6230"
server_key_iv: "90bc5b6fbb207e75b0396ddf520f2c1a"
server_key_version: "encryptionkey@0"
I20260927 14:51:29.228200 1292 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790520684602251-392-0/minicluster-data/ts-2/wal/instance:
uuid: "dcfd6b8afc3a459fa181d304fcf6054b"
format_stamp: "Formatted at 2026-09-27 14:51:29 on dist-test-slave-xk1g"
server_key: "32f94ed253c1caaaa1c41c74aacf6230"
server_key_iv: "90bc5b6fbb207e75b0396ddf520f2c1a"
server_key_version: "encryptionkey@0"
I20260927 14:51:29.231657 1292 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260927 14:51:29.233903 1307 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 14:51:29.235054 1292 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 14:51:29.235181 1292 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790520684602251-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790520684602251-392-0/minicluster-data/ts-2/wal
uuid: "dcfd6b8afc3a459fa181d304fcf6054b"
format_stamp: "Formatted at 2026-09-27 14:51:29 on dist-test-slave-xk1g"
server_key: "32f94ed253c1caaaa1c41c74aacf6230"
server_key_iv: "90bc5b6fbb207e75b0396ddf520f2c1a"
server_key_version: "encryptionkey@0"
I20260927 14:51:29.235280 1292 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790520684602251-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790520684602251-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790520684602251-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 14:51:29.253270 1292 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 14:51:29.256356 1292 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 14:51:29.256611 1292 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 14:51:29.257203 1292 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 14:51:29.257975 1292 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 14:51:29.258040 1292 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 14:51:29.258105 1292 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 14:51:29.258144 1292 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 14:51:29.266716 1292 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:39011
I20260927 14:51:29.266736 1420 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:39011 every 8 connection(s)
I20260927 14:51:29.267562 1292 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790520684602251-392-0/minicluster-data/ts-2/data/info.pb
I20260927 14:51:29.274838 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6936m5/build/debug/bin/kudu as pid 1292
I20260927 14:51:29.274977 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790520684602251-392-0/minicluster-data/ts-2/wal/instance
I20260927 14:51:29.275270 392 external_mini_cluster.cc:1428] Setting key 18d364f879ebe0808bee365e80e5481a
Sep 27 14:51:29 dist-test-slave-xk1g krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790520689, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 14:51:29.280687 1001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:51:29.269465 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:32941 (local address 127.0.98.62:34249)
0927 14:51:29.269597 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0927 14:51:29.269601 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 14:51:29.270413 (+ 812us) server_negotiation.cc:407] Connection header received
0927 14:51:29.271673 (+ 1260us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:51:29.271676 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:51:29.271722 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:51:29.271799 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:51:29.272925 (+ 1126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:51:29.273531 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:51:29.274639 (+ 1108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:51:29.274973 (+ 334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:51:29.275084 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:51:29.277644 (+ 2560us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:51:29.277647 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:51:29.277649 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 14:51:29.277672 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:51:29.278874 (+ 1202us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:51:29.279324 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:51:29.279326 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:51:29.279328 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:51:29.279402 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:51:29.279699 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:51:29.279701 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:51:29.279702 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:51:29.279930 (+ 228us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:51:29.280026 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0927 14:51:29.280477 (+ 451us) server_negotiation.cc:299] Negotiation successful
0927 14:51:29.280572 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260927 14:51:29.281630 1423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:51:29.269818 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34249 (local address 127.0.98.3:32941)
0927 14:51:29.270260 (+ 442us) negotiation.cc:107] Waiting for socket to connect
0927 14:51:29.270301 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0927 14:51:29.271478 (+ 1177us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:51:29.271956 (+ 478us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:51:29.271963 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:51:29.272312 (+ 349us) client_negotiation.cc:189] Negotiated authn=SASL
0927 14:51:29.272792 (+ 480us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:51:29.272804 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:51:29.273630 (+ 826us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:51:29.273633 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:51:29.274492 (+ 859us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:51:29.274500 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:51:29.275137 (+ 637us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:51:29.275140 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:51:29.275289 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:51:29.275918 (+ 629us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 14:51:29.275935 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 14:51:29.277504 (+ 1569us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 14:51:29.278971 (+ 1467us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 14:51:29.278977 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 14:51:29.278992 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0927 14:51:29.279235 (+ 243us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 14:51:29.279495 (+ 260us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 14:51:29.279497 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 14:51:29.279498 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 14:51:29.279620 (+ 122us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 14:51:29.280036 (+ 416us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 14:51:29.280041 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 14:51:29.280264 (+ 223us) client_negotiation.cc:769] Sending connection context
0927 14:51:29.280417 (+ 153us) client_negotiation.cc:240] Negotiation successful
0927 14:51:29.280672 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":266,"thread_start_us":133,"threads_started":1}
I20260927 14:51:29.282713 1421 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34249
I20260927 14:51:29.282932 1421 heartbeater.cc:464] Registering TS with master...
I20260927 14:51:29.283376 1421 heartbeater.cc:511] Master 127.0.98.62:34249 requested a full tablet report, sending...
I20260927 14:51:29.284317 962 ts_manager.cc:194] Registered new tserver with Master: dcfd6b8afc3a459fa181d304fcf6054b (127.0.98.3:39011)
I20260927 14:51:29.284873 962 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:32941
I20260927 14:51:29.288995 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 14:51:29.299284 1001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:51:29.292433 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48828 (local address 127.0.98.62:34249)
0927 14:51:29.292576 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0927 14:51:29.292581 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 14:51:29.292730 (+ 149us) server_negotiation.cc:407] Connection header received
0927 14:51:29.292835 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:51:29.292838 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:51:29.292879 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:51:29.292980 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:51:29.293639 (+ 659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:51:29.294175 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:51:29.294958 (+ 783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:51:29.295166 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:51:29.295251 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:51:29.296508 (+ 1257us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:51:29.296511 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:51:29.296513 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 14:51:29.296533 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:51:29.297857 (+ 1324us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:51:29.298222 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:51:29.298225 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:51:29.298226 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:51:29.298297 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:51:29.298509 (+ 212us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:51:29.298511 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:51:29.298512 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:51:29.298700 (+ 188us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:51:29.298799 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0927 14:51:29.298997 (+ 198us) server_negotiation.cc:299] Negotiation successful
0927 14:51:29.299091 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260927 14:51:29.304014 962 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48828:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 14:51:29.307209 962 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 14:51:29.322172 1437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:51:29.316409 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53372 (local address 127.0.98.3:39011)
0927 14:51:29.317322 (+ 913us) server_negotiation.cc:206] Beginning negotiation
0927 14:51:29.317336 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 14:51:29.317357 (+ 21us) server_negotiation.cc:407] Connection header received
0927 14:51:29.317427 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:51:29.317432 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:51:29.317644 (+ 212us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:51:29.317779 (+ 135us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 14:51:29.317945 (+ 166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:51:29.318667 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:51:29.320583 (+ 1916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:51:29.321410 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:51:29.321527 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:51:29.321678 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0927 14:51:29.321890 (+ 212us) server_negotiation.cc:299] Negotiation successful
0927 14:51:29.322004 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":817,"thread_start_us":78,"threads_started":1}
I20260927 14:51:29.322162 1433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:51:29.315547 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:33751 (local address 127.0.0.1:43342)
0927 14:51:29.315944 (+ 397us) negotiation.cc:107] Waiting for socket to connect
0927 14:51:29.316001 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0927 14:51:29.316123 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:51:29.316995 (+ 872us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:51:29.317001 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:51:29.317025 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 14:51:29.317236 (+ 211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:51:29.317244 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:51:29.318963 (+ 1719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:51:29.318966 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:51:29.320371 (+ 1405us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:51:29.320380 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:51:29.321746 (+ 1366us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:51:29.321749 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:51:29.321846 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:51:29.321869 (+ 23us) client_negotiation.cc:769] Sending connection context
0927 14:51:29.321936 (+ 67us) client_negotiation.cc:240] Negotiation successful
0927 14:51:29.322004 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":226,"mutex_wait_us":47,"thread_start_us":86,"threads_started":1}
I20260927 14:51:29.322167 1435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:51:29.316269 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:39011 (local address 127.0.0.1:53372)
0927 14:51:29.316956 (+ 687us) negotiation.cc:107] Waiting for socket to connect
0927 14:51:29.316968 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 14:51:29.317050 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:51:29.317757 (+ 707us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:51:29.317760 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:51:29.317775 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 14:51:29.317854 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:51:29.317859 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:51:29.318801 (+ 942us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:51:29.318807 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:51:29.320462 (+ 1655us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:51:29.320474 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:51:29.321540 (+ 1066us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:51:29.321544 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:51:29.321664 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:51:29.321708 (+ 44us) client_negotiation.cc:769] Sending connection context
0927 14:51:29.321830 (+ 122us) client_negotiation.cc:240] Negotiation successful
0927 14:51:29.321966 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":634,"mutex_wait_us":117,"thread_start_us":104,"threads_started":1}
I20260927 14:51:29.322818 1434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:51:29.315682 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43342 (local address 127.0.98.1:33751)
0927 14:51:29.316436 (+ 754us) server_negotiation.cc:206] Beginning negotiation
0927 14:51:29.316452 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0927 14:51:29.316478 (+ 26us) server_negotiation.cc:407] Connection header received
0927 14:51:29.316562 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:51:29.316569 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:51:29.316798 (+ 229us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:51:29.317455 (+ 657us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 14:51:29.317590 (+ 135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:51:29.318820 (+ 1230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:51:29.320504 (+ 1684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:51:29.321410 (+ 906us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:51:29.321527 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:51:29.321678 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0927 14:51:29.322557 (+ 879us) server_negotiation.cc:299] Negotiation successful
0927 14:51:29.322660 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":478,"thread_start_us":112,"threads_started":1}
I20260927 14:51:29.323017 1432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:51:29.315590 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:43027 (local address 127.0.0.1:46566)
0927 14:51:29.316592 (+ 1002us) negotiation.cc:107] Waiting for socket to connect
0927 14:51:29.316607 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 14:51:29.316690 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:51:29.317445 (+ 755us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:51:29.317448 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:51:29.317464 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 14:51:29.317557 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:51:29.317563 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:51:29.319126 (+ 1563us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:51:29.319129 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:51:29.320469 (+ 1340us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:51:29.320474 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:51:29.322427 (+ 1953us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:51:29.322430 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:51:29.322625 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:51:29.322653 (+ 28us) client_negotiation.cc:769] Sending connection context
0927 14:51:29.322716 (+ 63us) client_negotiation.cc:240] Negotiation successful
0927 14:51:29.322804 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":943,"mutex_wait_us":82,"thread_start_us":114,"threads_started":1}
I20260927 14:51:29.323585 1436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:51:29.316340 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46566 (local address 127.0.98.2:43027)
0927 14:51:29.316764 (+ 424us) server_negotiation.cc:206] Beginning negotiation
0927 14:51:29.316775 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 14:51:29.316797 (+ 22us) server_negotiation.cc:407] Connection header received
0927 14:51:29.316870 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:51:29.316876 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:51:29.317130 (+ 254us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:51:29.317279 (+ 149us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 14:51:29.317696 (+ 417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:51:29.318825 (+ 1129us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:51:29.321215 (+ 2390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:51:29.321936 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:51:29.322040 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:51:29.322204 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0927 14:51:29.323325 (+ 1121us) server_negotiation.cc:299] Negotiation successful
0927 14:51:29.323430 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":325,"thread_start_us":122,"threads_started":1}
I20260927 14:51:29.325796 1087 tablet_service.cc:1467] Processing CreateTablet for tablet e0517781dc8a49cc9fe359b8a036c1ad (DEFAULT_TABLE table=test-table [id=a2b3436a27924180bfcc2453151eee62]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 14:51:29.325845 1355 tablet_service.cc:1467] Processing CreateTablet for tablet e0517781dc8a49cc9fe359b8a036c1ad (DEFAULT_TABLE table=test-table [id=a2b3436a27924180bfcc2453151eee62]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 14:51:29.325963 1221 tablet_service.cc:1467] Processing CreateTablet for tablet e0517781dc8a49cc9fe359b8a036c1ad (DEFAULT_TABLE table=test-table [id=a2b3436a27924180bfcc2453151eee62]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 14:51:29.326848 1221 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e0517781dc8a49cc9fe359b8a036c1ad. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 14:51:29.326846 1355 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e0517781dc8a49cc9fe359b8a036c1ad. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 14:51:29.326848 1087 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e0517781dc8a49cc9fe359b8a036c1ad. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 14:51:29.332746 1438 tablet_bootstrap.cc:492] T e0517781dc8a49cc9fe359b8a036c1ad P ebe94da1eee14c91b9f5177e82ea5090: Bootstrap starting.
I20260927 14:51:29.334244 1439 tablet_bootstrap.cc:492] T e0517781dc8a49cc9fe359b8a036c1ad P 2bf38eb63cfc4236b6a6fd3261bad5c3: Bootstrap starting.
I20260927 14:51:29.335232 1438 tablet_bootstrap.cc:654] T e0517781dc8a49cc9fe359b8a036c1ad P ebe94da1eee14c91b9f5177e82ea5090: Neither blocks nor log segments found. Creating new log.
I20260927 14:51:29.335938 1438 log.cc:824] T e0517781dc8a49cc9fe359b8a036c1ad P ebe94da1eee14c91b9f5177e82ea5090: Log is configured to *not* fsync() on all Append() calls
I20260927 14:51:29.336084 1439 tablet_bootstrap.cc:654] T e0517781dc8a49cc9fe359b8a036c1ad P 2bf38eb63cfc4236b6a6fd3261bad5c3: Neither blocks nor log segments found. Creating new log.
I20260927 14:51:29.336547 1440 tablet_bootstrap.cc:492] T e0517781dc8a49cc9fe359b8a036c1ad P dcfd6b8afc3a459fa181d304fcf6054b: Bootstrap starting.
I20260927 14:51:29.337421 1439 log.cc:824] T e0517781dc8a49cc9fe359b8a036c1ad P 2bf38eb63cfc4236b6a6fd3261bad5c3: Log is configured to *not* fsync() on all Append() calls
I20260927 14:51:29.338097 1438 tablet_bootstrap.cc:492] T e0517781dc8a49cc9fe359b8a036c1ad P ebe94da1eee14c91b9f5177e82ea5090: No bootstrap required, opened a new log
I20260927 14:51:29.338366 1438 ts_tablet_manager.cc:1397] T e0517781dc8a49cc9fe359b8a036c1ad P ebe94da1eee14c91b9f5177e82ea5090: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.003s
I20260927 14:51:29.339004 1440 tablet_bootstrap.cc:654] T e0517781dc8a49cc9fe359b8a036c1ad P dcfd6b8afc3a459fa181d304fcf6054b: Neither blocks nor log segments found. Creating new log.
I20260927 14:51:29.339393 1439 tablet_bootstrap.cc:492] T e0517781dc8a49cc9fe359b8a036c1ad P 2bf38eb63cfc4236b6a6fd3261bad5c3: No bootstrap required, opened a new log
I20260927 14:51:29.339576 1439 ts_tablet_manager.cc:1397] T e0517781dc8a49cc9fe359b8a036c1ad P 2bf38eb63cfc4236b6a6fd3261bad5c3: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260927 14:51:29.340088 1440 log.cc:824] T e0517781dc8a49cc9fe359b8a036c1ad P dcfd6b8afc3a459fa181d304fcf6054b: Log is configured to *not* fsync() on all Append() calls
I20260927 14:51:29.341673 1440 tablet_bootstrap.cc:492] T e0517781dc8a49cc9fe359b8a036c1ad P dcfd6b8afc3a459fa181d304fcf6054b: No bootstrap required, opened a new log
I20260927 14:51:29.341856 1440 ts_tablet_manager.cc:1397] T e0517781dc8a49cc9fe359b8a036c1ad P dcfd6b8afc3a459fa181d304fcf6054b: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260927 14:51:29.343444 1438 raft_consensus.cc:348] T e0517781dc8a49cc9fe359b8a036c1ad P ebe94da1eee14c91b9f5177e82ea5090 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebe94da1eee14c91b9f5177e82ea5090" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33751 } } peers { permanent_uuid: "2bf38eb63cfc4236b6a6fd3261bad5c3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43027 } } peers { permanent_uuid: "dcfd6b8afc3a459fa181d304fcf6054b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39011 } }
I20260927 14:51:29.343734 1438 raft_consensus.cc:374] T e0517781dc8a49cc9fe359b8a036c1ad P ebe94da1eee14c91b9f5177e82ea5090 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 14:51:29.343788 1438 raft_consensus.cc:729] T e0517781dc8a49cc9fe359b8a036c1ad P ebe94da1eee14c91b9f5177e82ea5090 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ebe94da1eee14c91b9f5177e82ea5090, State: Initialized, Role: FOLLOWER
I20260927 14:51:29.343734 1439 raft_consensus.cc:348] T e0517781dc8a49cc9fe359b8a036c1ad P 2bf38eb63cfc4236b6a6fd3261bad5c3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebe94da1eee14c91b9f5177e82ea5090" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33751 } } peers { permanent_uuid: "2bf38eb63cfc4236b6a6fd3261bad5c3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43027 } } peers { permanent_uuid: "dcfd6b8afc3a459fa181d304fcf6054b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39011 } }
I20260927 14:51:29.343971 1439 raft_consensus.cc:374] T e0517781dc8a49cc9fe359b8a036c1ad P 2bf38eb63cfc4236b6a6fd3261bad5c3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 14:51:29.344017 1439 raft_consensus.cc:729] T e0517781dc8a49cc9fe359b8a036c1ad P 2bf38eb63cfc4236b6a6fd3261bad5c3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2bf38eb63cfc4236b6a6fd3261bad5c3, State: Initialized, Role: FOLLOWER
I20260927 14:51:29.344249 1438 consensus_queue.cc:260] T e0517781dc8a49cc9fe359b8a036c1ad P ebe94da1eee14c91b9f5177e82ea5090 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebe94da1eee14c91b9f5177e82ea5090" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33751 } } peers { permanent_uuid: "2bf38eb63cfc4236b6a6fd3261bad5c3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43027 } } peers { permanent_uuid: "dcfd6b8afc3a459fa181d304fcf6054b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39011 } }
I20260927 14:51:29.344421 1439 consensus_queue.cc:260] T e0517781dc8a49cc9fe359b8a036c1ad P 2bf38eb63cfc4236b6a6fd3261bad5c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebe94da1eee14c91b9f5177e82ea5090" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33751 } } peers { permanent_uuid: "2bf38eb63cfc4236b6a6fd3261bad5c3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43027 } } peers { permanent_uuid: "dcfd6b8afc3a459fa181d304fcf6054b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39011 } }
I20260927 14:51:29.345022 1440 raft_consensus.cc:348] T e0517781dc8a49cc9fe359b8a036c1ad P dcfd6b8afc3a459fa181d304fcf6054b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebe94da1eee14c91b9f5177e82ea5090" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33751 } } peers { permanent_uuid: "2bf38eb63cfc4236b6a6fd3261bad5c3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43027 } } peers { permanent_uuid: "dcfd6b8afc3a459fa181d304fcf6054b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39011 } }
I20260927 14:51:29.345264 1438 ts_tablet_manager.cc:1428] T e0517781dc8a49cc9fe359b8a036c1ad P ebe94da1eee14c91b9f5177e82ea5090: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260927 14:51:29.345264 1439 ts_tablet_manager.cc:1428] T e0517781dc8a49cc9fe359b8a036c1ad P 2bf38eb63cfc4236b6a6fd3261bad5c3: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260927 14:51:29.345279 1440 raft_consensus.cc:374] T e0517781dc8a49cc9fe359b8a036c1ad P dcfd6b8afc3a459fa181d304fcf6054b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 14:51:29.345340 1440 raft_consensus.cc:729] T e0517781dc8a49cc9fe359b8a036c1ad P dcfd6b8afc3a459fa181d304fcf6054b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dcfd6b8afc3a459fa181d304fcf6054b, State: Initialized, Role: FOLLOWER
I20260927 14:51:29.345642 1153 heartbeater.cc:503] Master 127.0.98.62:34249 was elected leader, sending a full tablet report...
I20260927 14:51:29.345773 1440 consensus_queue.cc:260] T e0517781dc8a49cc9fe359b8a036c1ad P dcfd6b8afc3a459fa181d304fcf6054b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebe94da1eee14c91b9f5177e82ea5090" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33751 } } peers { permanent_uuid: "2bf38eb63cfc4236b6a6fd3261bad5c3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43027 } } peers { permanent_uuid: "dcfd6b8afc3a459fa181d304fcf6054b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39011 } }
I20260927 14:51:29.346383 1421 heartbeater.cc:503] Master 127.0.98.62:34249 was elected leader, sending a full tablet report...
I20260927 14:51:29.346647 1287 heartbeater.cc:503] Master 127.0.98.62:34249 was elected leader, sending a full tablet report...
I20260927 14:51:29.346728 1440 ts_tablet_manager.cc:1428] T e0517781dc8a49cc9fe359b8a036c1ad P dcfd6b8afc3a459fa181d304fcf6054b: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260927 14:51:29.453397 1446 raft_consensus.cc:482] T e0517781dc8a49cc9fe359b8a036c1ad P dcfd6b8afc3a459fa181d304fcf6054b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 14:51:29.453622 1446 raft_consensus.cc:504] T e0517781dc8a49cc9fe359b8a036c1ad P dcfd6b8afc3a459fa181d304fcf6054b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebe94da1eee14c91b9f5177e82ea5090" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33751 } } peers { permanent_uuid: "2bf38eb63cfc4236b6a6fd3261bad5c3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43027 } } peers { permanent_uuid: "dcfd6b8afc3a459fa181d304fcf6054b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39011 } }
I20260927 14:51:29.454802 1446 leader_election.cc:288] T e0517781dc8a49cc9fe359b8a036c1ad P dcfd6b8afc3a459fa181d304fcf6054b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ebe94da1eee14c91b9f5177e82ea5090 (127.0.98.1:33751), 2bf38eb63cfc4236b6a6fd3261bad5c3 (127.0.98.2:43027)
I20260927 14:51:29.460016 1436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:51:29.455257 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:45241 (local address 127.0.98.2:43027)
0927 14:51:29.455399 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0927 14:51:29.455420 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0927 14:51:29.455518 (+ 98us) server_negotiation.cc:407] Connection header received
0927 14:51:29.455601 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:51:29.455604 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:51:29.455659 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:51:29.455739 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 14:51:29.456000 (+ 261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:51:29.456801 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:51:29.458732 (+ 1931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:51:29.459392 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:51:29.459532 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:51:29.459582 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0927 14:51:29.459809 (+ 227us) server_negotiation.cc:299] Negotiation successful
0927 14:51:29.459880 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260927 14:51:29.460016 1434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:51:29.455106 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:55113 (local address 127.0.98.1:33751)
0927 14:51:29.455224 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0927 14:51:29.455230 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 14:51:29.455375 (+ 145us) server_negotiation.cc:407] Connection header received
0927 14:51:29.455453 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:51:29.455457 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:51:29.455550 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:51:29.455637 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 14:51:29.455996 (+ 359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:51:29.456768 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:51:29.458732 (+ 1964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:51:29.459408 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:51:29.459514 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:51:29.459580 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0927 14:51:29.459809 (+ 229us) server_negotiation.cc:299] Negotiation successful
0927 14:51:29.459880 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260927 14:51:29.460089 1423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:51:29.455023 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:33751 (local address 127.0.98.3:55113)
0927 14:51:29.455222 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0927 14:51:29.455238 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 14:51:29.455338 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:51:29.455665 (+ 327us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:51:29.455667 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:51:29.455679 (+ 12us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 14:51:29.455820 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:51:29.455827 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:51:29.456903 (+ 1076us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:51:29.456906 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:51:29.458587 (+ 1681us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:51:29.458599 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:51:29.459520 (+ 921us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:51:29.459524 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:51:29.459677 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:51:29.459706 (+ 29us) client_negotiation.cc:769] Sending connection context
0927 14:51:29.459783 (+ 77us) client_negotiation.cc:240] Negotiation successful
0927 14:51:29.459923 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"mutex_wait_us":291}
I20260927 14:51:29.460098 1447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:51:29.455069 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:43027 (local address 127.0.98.3:45241)
0927 14:51:29.455419 (+ 350us) negotiation.cc:107] Waiting for socket to connect
0927 14:51:29.455433 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 14:51:29.455512 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:51:29.455746 (+ 234us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:51:29.455748 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:51:29.455760 (+ 12us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 14:51:29.455877 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:51:29.455885 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:51:29.456903 (+ 1018us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:51:29.456906 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:51:29.458587 (+ 1681us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:51:29.458599 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:51:29.459544 (+ 945us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:51:29.459547 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:51:29.459672 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:51:29.459703 (+ 31us) client_negotiation.cc:769] Sending connection context
0927 14:51:29.459783 (+ 80us) client_negotiation.cc:240] Negotiation successful
0927 14:51:29.459923 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":280,"mutex_wait_us":209,"thread_start_us":136,"threads_started":1}
I20260927 14:51:29.460811 1107 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0517781dc8a49cc9fe359b8a036c1ad" candidate_uuid: "dcfd6b8afc3a459fa181d304fcf6054b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ebe94da1eee14c91b9f5177e82ea5090" is_pre_election: true
I20260927 14:51:29.460810 1241 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0517781dc8a49cc9fe359b8a036c1ad" candidate_uuid: "dcfd6b8afc3a459fa181d304fcf6054b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2bf38eb63cfc4236b6a6fd3261bad5c3" is_pre_election: true
I20260927 14:51:29.461167 1107 raft_consensus.cc:2454] T e0517781dc8a49cc9fe359b8a036c1ad P ebe94da1eee14c91b9f5177e82ea5090 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dcfd6b8afc3a459fa181d304fcf6054b in term 0.
I20260927 14:51:29.461167 1241 raft_consensus.cc:2454] T e0517781dc8a49cc9fe359b8a036c1ad P 2bf38eb63cfc4236b6a6fd3261bad5c3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dcfd6b8afc3a459fa181d304fcf6054b in term 0.
I20260927 14:51:29.461671 1309 leader_election.cc:302] T e0517781dc8a49cc9fe359b8a036c1ad P dcfd6b8afc3a459fa181d304fcf6054b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2bf38eb63cfc4236b6a6fd3261bad5c3, dcfd6b8afc3a459fa181d304fcf6054b; no voters:
I20260927 14:51:29.461952 1446 raft_consensus.cc:2781] T e0517781dc8a49cc9fe359b8a036c1ad P dcfd6b8afc3a459fa181d304fcf6054b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 14:51:29.462038 1446 raft_consensus.cc:482] T e0517781dc8a49cc9fe359b8a036c1ad P dcfd6b8afc3a459fa181d304fcf6054b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 14:51:29.462086 1446 raft_consensus.cc:3037] T e0517781dc8a49cc9fe359b8a036c1ad P dcfd6b8afc3a459fa181d304fcf6054b [term 0 FOLLOWER]: Advancing to term 1
I20260927 14:51:29.463161 1446 raft_consensus.cc:504] T e0517781dc8a49cc9fe359b8a036c1ad P dcfd6b8afc3a459fa181d304fcf6054b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebe94da1eee14c91b9f5177e82ea5090" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33751 } } peers { permanent_uuid: "2bf38eb63cfc4236b6a6fd3261bad5c3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43027 } } peers { permanent_uuid: "dcfd6b8afc3a459fa181d304fcf6054b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39011 } }
I20260927 14:51:29.463523 1446 leader_election.cc:288] T e0517781dc8a49cc9fe359b8a036c1ad P dcfd6b8afc3a459fa181d304fcf6054b [CANDIDATE]: Term 1 election: Requested vote from peers ebe94da1eee14c91b9f5177e82ea5090 (127.0.98.1:33751), 2bf38eb63cfc4236b6a6fd3261bad5c3 (127.0.98.2:43027)
I20260927 14:51:29.463902 1241 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0517781dc8a49cc9fe359b8a036c1ad" candidate_uuid: "dcfd6b8afc3a459fa181d304fcf6054b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2bf38eb63cfc4236b6a6fd3261bad5c3"
I20260927 14:51:29.463939 1107 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0517781dc8a49cc9fe359b8a036c1ad" candidate_uuid: "dcfd6b8afc3a459fa181d304fcf6054b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ebe94da1eee14c91b9f5177e82ea5090"
I20260927 14:51:29.464057 1241 raft_consensus.cc:3037] T e0517781dc8a49cc9fe359b8a036c1ad P 2bf38eb63cfc4236b6a6fd3261bad5c3 [term 0 FOLLOWER]: Advancing to term 1
I20260927 14:51:29.464097 1107 raft_consensus.cc:3037] T e0517781dc8a49cc9fe359b8a036c1ad P ebe94da1eee14c91b9f5177e82ea5090 [term 0 FOLLOWER]: Advancing to term 1
I20260927 14:51:29.465306 1107 raft_consensus.cc:2454] T e0517781dc8a49cc9fe359b8a036c1ad P ebe94da1eee14c91b9f5177e82ea5090 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dcfd6b8afc3a459fa181d304fcf6054b in term 1.
I20260927 14:51:29.465457 1241 raft_consensus.cc:2454] T e0517781dc8a49cc9fe359b8a036c1ad P 2bf38eb63cfc4236b6a6fd3261bad5c3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dcfd6b8afc3a459fa181d304fcf6054b in term 1.
I20260927 14:51:29.465623 1311 leader_election.cc:302] T e0517781dc8a49cc9fe359b8a036c1ad P dcfd6b8afc3a459fa181d304fcf6054b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: dcfd6b8afc3a459fa181d304fcf6054b, ebe94da1eee14c91b9f5177e82ea5090; no voters:
I20260927 14:51:29.465826 1446 raft_consensus.cc:2781] T e0517781dc8a49cc9fe359b8a036c1ad P dcfd6b8afc3a459fa181d304fcf6054b [term 1 FOLLOWER]: Leader election won for term 1
I20260927 14:51:29.466073 1446 raft_consensus.cc:686] T e0517781dc8a49cc9fe359b8a036c1ad P dcfd6b8afc3a459fa181d304fcf6054b [term 1 LEADER]: Becoming Leader. State: Replica: dcfd6b8afc3a459fa181d304fcf6054b, State: Running, Role: LEADER
I20260927 14:51:29.466389 1446 consensus_queue.cc:237] T e0517781dc8a49cc9fe359b8a036c1ad P dcfd6b8afc3a459fa181d304fcf6054b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebe94da1eee14c91b9f5177e82ea5090" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33751 } } peers { permanent_uuid: "2bf38eb63cfc4236b6a6fd3261bad5c3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43027 } } peers { permanent_uuid: "dcfd6b8afc3a459fa181d304fcf6054b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39011 } }
I20260927 14:51:29.469449 961 catalog_manager.cc:5495] T e0517781dc8a49cc9fe359b8a036c1ad P dcfd6b8afc3a459fa181d304fcf6054b reported cstate change: term changed from 0 to 1, leader changed from <none> to dcfd6b8afc3a459fa181d304fcf6054b (127.0.98.3). New cstate: current_term: 1 leader_uuid: "dcfd6b8afc3a459fa181d304fcf6054b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebe94da1eee14c91b9f5177e82ea5090" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33751 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2bf38eb63cfc4236b6a6fd3261bad5c3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43027 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dcfd6b8afc3a459fa181d304fcf6054b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39011 } health_report { overall_health: HEALTHY } } }
Sep 27 14:51:29 dist-test-slave-xk1g krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790520688, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260927 14:51:29.512909 1434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:51:29.503194 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43352 (local address 127.0.98.1:33751)
0927 14:51:29.503340 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0927 14:51:29.503345 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 14:51:29.503514 (+ 169us) server_negotiation.cc:407] Connection header received
0927 14:51:29.503639 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:51:29.503641 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:51:29.503695 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:51:29.503760 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:51:29.504418 (+ 658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:51:29.504926 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:51:29.505820 (+ 894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:51:29.506076 (+ 256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:51:29.506168 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:51:29.508798 (+ 2630us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:51:29.508810 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:51:29.508825 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 14:51:29.508849 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:51:29.510878 (+ 2029us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:51:29.511343 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:51:29.511349 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:51:29.511350 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:51:29.511430 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:51:29.511883 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:51:29.511885 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:51:29.511886 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:51:29.512162 (+ 276us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:51:29.512333 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0927 14:51:29.512589 (+ 256us) server_negotiation.cc:299] Negotiation successful
0927 14:51:29.512693 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
W20260927 14:51:29.518477 1422 tablet.cc:2369] T e0517781dc8a49cc9fe359b8a036c1ad P dcfd6b8afc3a459fa181d304fcf6054b: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 27 14:51:29 dist-test-slave-xk1g krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790520689, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 27 14:51:29 dist-test-slave-xk1g krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790520689, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260927 14:51:29.539966 1434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:51:29.531176 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43360 (local address 127.0.98.1:33751)
0927 14:51:29.531296 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0927 14:51:29.531302 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 14:51:29.531386 (+ 84us) server_negotiation.cc:407] Connection header received
0927 14:51:29.531562 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:51:29.531565 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:51:29.531614 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:51:29.531699 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:51:29.532364 (+ 665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:51:29.532891 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:51:29.533811 (+ 920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:51:29.534067 (+ 256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:51:29.534175 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:51:29.536782 (+ 2607us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:51:29.536785 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:51:29.536787 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 14:51:29.536811 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:51:29.538308 (+ 1497us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:51:29.538704 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:51:29.538708 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:51:29.538710 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:51:29.538791 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:51:29.539070 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:51:29.539075 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:51:29.539077 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:51:29.539267 (+ 190us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:51:29.539393 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0927 14:51:29.539663 (+ 270us) server_negotiation.cc:299] Negotiation successful
0927 14:51:29.539793 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
Sep 27 14:51:29 dist-test-slave-xk1g krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790520689, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
W20260927 14:51:29.553457 1288 tablet.cc:2369] T e0517781dc8a49cc9fe359b8a036c1ad P 2bf38eb63cfc4236b6a6fd3261bad5c3: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 27 14:51:29 dist-test-slave-xk1g krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790520689, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260927 14:51:29.565737 1434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:51:29.557823 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43376 (local address 127.0.98.1:33751)
0927 14:51:29.557991 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0927 14:51:29.557996 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 14:51:29.558127 (+ 131us) server_negotiation.cc:407] Connection header received
0927 14:51:29.558281 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:51:29.558285 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:51:29.558332 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:51:29.558417 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:51:29.559080 (+ 663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:51:29.559578 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:51:29.560401 (+ 823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:51:29.560685 (+ 284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:51:29.560789 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:51:29.563081 (+ 2292us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:51:29.563084 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:51:29.563087 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 14:51:29.563107 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:51:29.564297 (+ 1190us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:51:29.564714 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:51:29.564717 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:51:29.564718 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:51:29.564787 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:51:29.565027 (+ 240us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:51:29.565029 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:51:29.565030 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:51:29.565222 (+ 192us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:51:29.565320 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0927 14:51:29.565529 (+ 209us) server_negotiation.cc:299] Negotiation successful
0927 14:51:29.565621 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
W20260927 14:51:29.567014 1067 tablet_service.cc:2908] Rejecting scan request for tablet e0517781dc8a49cc9fe359b8a036c1ad: Uninitialized: clean time has not yet been initialized
I20260927 14:51:29.568943 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6936m5/build/debug/bin/kudu with pid 1024
I20260927 14:51:29.574841 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6936m5/build/debug/bin/kudu with pid 1158
I20260927 14:51:29.580080 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6936m5/build/debug/bin/kudu with pid 1292
I20260927 14:51:29.585424 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6936m5/build/debug/bin/kudu with pid 930
2026-09-27T14:51:29Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (2779 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790520684602251-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 14:51:29 dist-test-slave-xk1g krb5kdc[1462](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 14:51:29 dist-test-slave-xk1g krb5kdc[1462](info): set up 2 sockets
Sep 27 14:51:29 dist-test-slave-xk1g krb5kdc[1462](info): commencing operation
krb5kdc: starting...
W20260927 14:51:30.696580 392 mini_kdc.cc:121] Time spent starting KDC: real 1.089s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790520684602251-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790520684602251-392-0/krb5kdc/test-user.keytab.
Sep 27 14:51:30 dist-test-slave-xk1g krb5kdc[1462](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790520690, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T14:51:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T14:51:30Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790520684602251-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790520684602251-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 14:51:30.816256 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6936m5/build/debug/bin/kudu
/tmp/dist-test-task6936m5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790520684602251-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790520684602251-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790520684602251-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790520684602251-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40621
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790520684602251-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40231
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:40621
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790520684602251-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790520684602251-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790520684602251-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 14:51:30.920023 1475 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 14:51:30.920326 1475 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 14:51:30.920396 1475 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 14:51:30.924347 1475 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 14:51:30.924428 1475 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260927 14:51:30.924459 1475 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 14:51:30.924518 1475 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 14:51:30.924558 1475 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 14:51:30.924594 1475 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 14:51:30.929679 1475 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40231
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790520684602251-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790520684602251-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:40621
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790520684602251-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40621
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790520684602251-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1475
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790520684602251-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:15:17 UTC on ccycloud.quasar-ledxfe.root.comops.site
build id 4676134
I20260927 14:51:30.930681 1475 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 14:51:30.931686 1475 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 14:51:30.939877 1480 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 14:51:30.939914 1481 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 14:51:30.939952 1475 server_base.cc:1034] running on GCE node
W20260927 14:51:30.939894 1483 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 14:51:30.940660 1475 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 14:51:30.941715 1475 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 14:51:30.942888 1475 hybrid_clock.cc:648] HybridClock initialized: now 1790520690942870 us; error 28 us; skew 500 ppm
Sep 27 14:51:30 dist-test-slave-xk1g krb5kdc[1462](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790520690, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 14:51:30.948674 1475 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 14:51:30.949556 1475 webserver.cc:460] Webserver started at http://127.0.98.62:34315/ using document root <none> and password file <none>
I20260927 14:51:30.950050 1475 fs_manager.cc:359] Metadata directory not provided
I20260927 14:51:30.950105 1475 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 14:51:30.950260 1475 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 14:51:30.952157 1475 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790520684602251-392-0/minicluster-data/master-0/data/instance:
uuid: "a03aa1549f69443084f0c6a8c1b06bd6"
format_stamp: "Formatted at 2026-09-27 14:51:30 on dist-test-slave-xk1g"
server_key: "e5c27df040ba86339911b64f647627e8"
server_key_iv: "886e8fadb170b372473c914ec591db13"
server_key_version: "encryptionkey@0"
I20260927 14:51:30.952670 1475 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790520684602251-392-0/minicluster-data/master-0/wal/instance:
uuid: "a03aa1549f69443084f0c6a8c1b06bd6"
format_stamp: "Formatted at 2026-09-27 14:51:30 on dist-test-slave-xk1g"
server_key: "e5c27df040ba86339911b64f647627e8"
server_key_iv: "886e8fadb170b372473c914ec591db13"
server_key_version: "encryptionkey@0"
I20260927 14:51:30.956133 1475 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260927 14:51:30.958346 1490 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 14:51:30.959476 1475 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 14:51:30.959589 1475 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790520684602251-392-0/minicluster-data/master-0/data,/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790520684602251-392-0/minicluster-data/master-0/wal
uuid: "a03aa1549f69443084f0c6a8c1b06bd6"
format_stamp: "Formatted at 2026-09-27 14:51:30 on dist-test-slave-xk1g"
server_key: "e5c27df040ba86339911b64f647627e8"
server_key_iv: "886e8fadb170b372473c914ec591db13"
server_key_version: "encryptionkey@0"
I20260927 14:51:30.959679 1475 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790520684602251-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790520684602251-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790520684602251-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 14:51:30.975189 1475 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 14:51:30.977973 1475 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 14:51:30.978159 1475 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 14:51:30.986235 1475 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:40621
I20260927 14:51:30.986254 1552 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:40621 every 8 connection(s)
I20260927 14:51:30.987205 1475 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790520684602251-392-0/minicluster-data/master-0/data/info.pb
I20260927 14:51:30.989951 1553 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 14:51:30.991722 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6936m5/build/debug/bin/kudu as pid 1475
I20260927 14:51:30.991807 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790520684602251-392-0/minicluster-data/master-0/wal/instance
I20260927 14:51:30.992071 392 external_mini_cluster.cc:1428] Setting key cfe857da6a90ac19b33b9c654e5c0dc2
I20260927 14:51:30.996618 1553 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a03aa1549f69443084f0c6a8c1b06bd6: Bootstrap starting.
I20260927 14:51:30.999157 1553 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a03aa1549f69443084f0c6a8c1b06bd6: Neither blocks nor log segments found. Creating new log.
I20260927 14:51:30.999923 1553 log.cc:824] T 00000000000000000000000000000000 P a03aa1549f69443084f0c6a8c1b06bd6: Log is configured to *not* fsync() on all Append() calls
Sep 27 14:51:31 dist-test-slave-xk1g krb5kdc[1462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790520690, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 14:51:31.002246 1553 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a03aa1549f69443084f0c6a8c1b06bd6: No bootstrap required, opened a new log
I20260927 14:51:31.005553 1553 raft_consensus.cc:348] T 00000000000000000000000000000000 P a03aa1549f69443084f0c6a8c1b06bd6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a03aa1549f69443084f0c6a8c1b06bd6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40621 } }
I20260927 14:51:31.005797 1553 raft_consensus.cc:374] T 00000000000000000000000000000000 P a03aa1549f69443084f0c6a8c1b06bd6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 14:51:31.005882 1553 raft_consensus.cc:729] T 00000000000000000000000000000000 P a03aa1549f69443084f0c6a8c1b06bd6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a03aa1549f69443084f0c6a8c1b06bd6, State: Initialized, Role: FOLLOWER
I20260927 14:51:31.006323 1553 consensus_queue.cc:260] T 00000000000000000000000000000000 P a03aa1549f69443084f0c6a8c1b06bd6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a03aa1549f69443084f0c6a8c1b06bd6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40621 } }
I20260927 14:51:31.006444 1553 raft_consensus.cc:388] T 00000000000000000000000000000000 P a03aa1549f69443084f0c6a8c1b06bd6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 14:51:31.006524 1553 raft_consensus.cc:482] T 00000000000000000000000000000000 P a03aa1549f69443084f0c6a8c1b06bd6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 14:51:31.006661 1553 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a03aa1549f69443084f0c6a8c1b06bd6 [term 0 FOLLOWER]: Advancing to term 1
I20260927 14:51:31.007630 1553 raft_consensus.cc:504] T 00000000000000000000000000000000 P a03aa1549f69443084f0c6a8c1b06bd6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a03aa1549f69443084f0c6a8c1b06bd6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40621 } }
I20260927 14:51:31.007956 1553 leader_election.cc:302] T 00000000000000000000000000000000 P a03aa1549f69443084f0c6a8c1b06bd6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a03aa1549f69443084f0c6a8c1b06bd6; no voters:
I20260927 14:51:31.008006 1556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:51:30.993496 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53354 (local address 127.0.98.62:40621)
0927 14:51:30.993886 (+ 390us) server_negotiation.cc:206] Beginning negotiation
0927 14:51:30.993904 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0927 14:51:30.993948 (+ 44us) server_negotiation.cc:407] Connection header received
0927 14:51:30.994759 (+ 811us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:51:30.994782 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:51:30.995233 (+ 451us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:51:30.995619 (+ 386us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:51:30.996956 (+ 1337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:51:30.997995 (+ 1039us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:51:30.999126 (+ 1131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:51:30.999686 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:51:30.999843 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:51:31.002702 (+ 2859us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:51:31.002707 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:51:31.002720 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 14:51:31.002745 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:51:31.005032 (+ 2287us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:51:31.005502 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:51:31.005508 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:51:31.005514 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:51:31.005625 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:51:31.005952 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:51:31.005957 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:51:31.005960 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:51:31.006430 (+ 470us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:51:31.006639 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0927 14:51:31.006947 (+ 308us) server_negotiation.cc:299] Negotiation successful
0927 14:51:31.007227 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":250,"thread_start_us":102,"threads_started":1}
I20260927 14:51:31.008312 1553 leader_election.cc:288] T 00000000000000000000000000000000 P a03aa1549f69443084f0c6a8c1b06bd6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 14:51:31.008414 1558 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a03aa1549f69443084f0c6a8c1b06bd6 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 14:51:31.008621 1558 raft_consensus.cc:686] T 00000000000000000000000000000000 P a03aa1549f69443084f0c6a8c1b06bd6 [term 1 LEADER]: Becoming Leader. State: Replica: a03aa1549f69443084f0c6a8c1b06bd6, State: Running, Role: LEADER
I20260927 14:51:31.008919 1558 consensus_queue.cc:237] T 00000000000000000000000000000000 P a03aa1549f69443084f0c6a8c1b06bd6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a03aa1549f69443084f0c6a8c1b06bd6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40621 } }
I20260927 14:51:31.009521 1553 sys_catalog.cc:564] T 00000000000000000000000000000000 P a03aa1549f69443084f0c6a8c1b06bd6 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 14:51:31.010521 1559 sys_catalog.cc:455] T 00000000000000000000000000000000 P a03aa1549f69443084f0c6a8c1b06bd6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a03aa1549f69443084f0c6a8c1b06bd6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a03aa1549f69443084f0c6a8c1b06bd6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40621 } } }
I20260927 14:51:31.010651 1559 sys_catalog.cc:458] T 00000000000000000000000000000000 P a03aa1549f69443084f0c6a8c1b06bd6 [sys.catalog]: This master's current role is: LEADER
I20260927 14:51:31.010977 1560 sys_catalog.cc:455] T 00000000000000000000000000000000 P a03aa1549f69443084f0c6a8c1b06bd6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a03aa1549f69443084f0c6a8c1b06bd6. Latest consensus state: current_term: 1 leader_uuid: "a03aa1549f69443084f0c6a8c1b06bd6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a03aa1549f69443084f0c6a8c1b06bd6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40621 } } }
I20260927 14:51:31.011072 1560 sys_catalog.cc:458] T 00000000000000000000000000000000 P a03aa1549f69443084f0c6a8c1b06bd6 [sys.catalog]: This master's current role is: LEADER
W20260927 14:51:31.014472 1573 catalog_manager.cc:1558] T 00000000000000000000000000000000 P a03aa1549f69443084f0c6a8c1b06bd6: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 14:51:31.014544 1573 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 14:51:31.014648 1567 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 14:51:31.015322 1567 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 14:51:31.020032 1567 catalog_manager.cc:1347] Generated new cluster ID: 941f7edd981f4bcaa46562fb60429f5c
I20260927 14:51:31.020097 1567 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 14:51:31.041210 1567 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 14:51:31.042500 1567 catalog_manager.cc:1504] Loading token signing keys...
I20260927 14:51:31.051493 1567 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a03aa1549f69443084f0c6a8c1b06bd6: Generated new TSK 0
I20260927 14:51:31.052167 1567 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790520684602251-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790520684602251-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260927 14:51:31.084137 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6936m5/build/debug/bin/kudu
/tmp/dist-test-task6936m5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790520684602251-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790520684602251-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790520684602251-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790520684602251-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40621
--keytab_file=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790520684602251-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40231
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790520684602251-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790520684602251-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790520684602251-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 14:51:31.216259 1579 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 14:51:31.216524 1579 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 14:51:31.216573 1579 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 14:51:31.220503 1579 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 14:51:31.220558 1579 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 14:51:31.220597 1579 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 14:51:31.220628 1579 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 14:51:31.225756 1579 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40231
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790520684602251-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790520684602251-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790520684602251-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790520684602251-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:40621
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1579
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790520684602251-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:15:17 UTC on ccycloud.quasar-ledxfe.root.comops.site
build id 4676134
I20260927 14:51:31.226737 1579 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 14:51:31.227669 1579 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 14:51:31.236908 1585 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 14:51:31.236913 1587 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 14:51:31.236913 1584 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 14:51:31.236949 1579 server_base.cc:1034] running on GCE node
I20260927 14:51:31.237520 1579 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 14:51:31.238111 1579 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 14:51:31.239300 1579 hybrid_clock.cc:648] HybridClock initialized: now 1790520691239266 us; error 46 us; skew 500 ppm
Sep 27 14:51:31 dist-test-slave-xk1g krb5kdc[1462](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790520691, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 14:51:31.242066 1579 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 14:51:31.242871 1579 webserver.cc:460] Webserver started at http://127.0.98.1:44683/ using document root <none> and password file <none>
I20260927 14:51:31.243396 1579 fs_manager.cc:359] Metadata directory not provided
I20260927 14:51:31.243481 1579 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 14:51:31.243705 1579 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 14:51:31.245714 1579 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790520684602251-392-0/minicluster-data/ts-0/data/instance:
uuid: "9c32f7b24fbc4b148527a0b1fedf8db8"
format_stamp: "Formatted at 2026-09-27 14:51:31 on dist-test-slave-xk1g"
server_key: "bb5fd141e859b1288f4f29880f79d172"
server_key_iv: "1888e9358c98ddbd567385b92c98ee9b"
server_key_version: "encryptionkey@0"
I20260927 14:51:31.246254 1579 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790520684602251-392-0/minicluster-data/ts-0/wal/instance:
uuid: "9c32f7b24fbc4b148527a0b1fedf8db8"
format_stamp: "Formatted at 2026-09-27 14:51:31 on dist-test-slave-xk1g"
server_key: "bb5fd141e859b1288f4f29880f79d172"
server_key_iv: "1888e9358c98ddbd567385b92c98ee9b"
server_key_version: "encryptionkey@0"
I20260927 14:51:31.249749 1579 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 14:51:31.251967 1594 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 14:51:31.253294 1579 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 14:51:31.253437 1579 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790520684602251-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790520684602251-392-0/minicluster-data/ts-0/wal
uuid: "9c32f7b24fbc4b148527a0b1fedf8db8"
format_stamp: "Formatted at 2026-09-27 14:51:31 on dist-test-slave-xk1g"
server_key: "bb5fd141e859b1288f4f29880f79d172"
server_key_iv: "1888e9358c98ddbd567385b92c98ee9b"
server_key_version: "encryptionkey@0"
I20260927 14:51:31.253547 1579 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790520684602251-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790520684602251-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790520684602251-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 14:51:31.272156 1579 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 14:51:31.275866 1579 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 14:51:31.276094 1579 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 14:51:31.277699 1579 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 14:51:31.277778 1579 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 14:51:31.277855 1579 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 14:51:31.277899 1579 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 27 14:51:31 dist-test-slave-xk1g krb5kdc[1462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790520691, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 14:51:31.289772 1579 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:43271
I20260927 14:51:31.289991 1709 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:43271 every 8 connection(s)
I20260927 14:51:31.290962 1579 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790520684602251-392-0/minicluster-data/ts-0/data/info.pb
I20260927 14:51:31.292703 1603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:51:31.278898 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40621 (local address 127.0.98.1:59863)
0927 14:51:31.279363 (+ 465us) negotiation.cc:107] Waiting for socket to connect
0927 14:51:31.279413 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0927 14:51:31.280352 (+ 939us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:51:31.281219 (+ 867us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:51:31.281228 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:51:31.281604 (+ 376us) client_negotiation.cc:189] Negotiated authn=SASL
0927 14:51:31.282063 (+ 459us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:51:31.282077 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:51:31.282962 (+ 885us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:51:31.282965 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:51:31.284254 (+ 1289us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:51:31.284267 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:51:31.284776 (+ 509us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:51:31.284780 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:51:31.284962 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:51:31.285656 (+ 694us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 14:51:31.285675 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0927 14:51:31.287802 (+ 2127us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 14:51:31.289736 (+ 1934us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 14:51:31.289742 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 14:51:31.289756 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0927 14:51:31.290175 (+ 419us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 14:51:31.290502 (+ 327us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 14:51:31.290504 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 14:51:31.290506 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 14:51:31.290674 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 14:51:31.291212 (+ 538us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 14:51:31.291218 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 14:51:31.291505 (+ 287us) client_negotiation.cc:769] Sending connection context
0927 14:51:31.291705 (+ 200us) client_negotiation.cc:240] Negotiation successful
0927 14:51:31.292004 (+ 299us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":259,"thread_start_us":138,"threads_started":1}
I20260927 14:51:31.293322 1556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:51:31.278535 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:59863 (local address 127.0.98.62:40621)
0927 14:51:31.278708 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0927 14:51:31.278716 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 14:51:31.279544 (+ 828us) server_negotiation.cc:407] Connection header received
0927 14:51:31.280612 (+ 1068us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:51:31.280616 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:51:31.280679 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:51:31.280739 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:51:31.282206 (+ 1467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:51:31.282844 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:51:31.284378 (+ 1534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:51:31.284623 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:51:31.284739 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:51:31.287970 (+ 3231us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:51:31.287974 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:51:31.287978 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 14:51:31.288009 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:51:31.289627 (+ 1618us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:51:31.290280 (+ 653us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:51:31.290284 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:51:31.290286 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:51:31.290391 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:51:31.290772 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:51:31.290777 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:51:31.290778 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:51:31.291059 (+ 281us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:51:31.291170 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0927 14:51:31.293091 (+ 1921us) server_negotiation.cc:299] Negotiation successful
0927 14:51:31.293212 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260927 14:51:31.294267 1710 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40621
I20260927 14:51:31.294404 1710 heartbeater.cc:464] Registering TS with master...
I20260927 14:51:31.294979 1710 heartbeater.cc:511] Master 127.0.98.62:40621 requested a full tablet report, sending...
I20260927 14:51:31.296468 1507 ts_manager.cc:194] Registered new tserver with Master: 9c32f7b24fbc4b148527a0b1fedf8db8 (127.0.98.1:43271)
I20260927 14:51:31.297663 1507 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:59863
I20260927 14:51:31.300745 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6936m5/build/debug/bin/kudu as pid 1579
I20260927 14:51:31.300881 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790520684602251-392-0/minicluster-data/ts-0/wal/instance
I20260927 14:51:31.301232 392 external_mini_cluster.cc:1428] Setting key 9175fb6bc2739b02a56503a22553fb58
I20260927 14:51:31.303417 1556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:51:31.296549 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:35931 (local address 127.0.98.62:40621)
0927 14:51:31.296667 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0927 14:51:31.296671 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 14:51:31.296823 (+ 152us) server_negotiation.cc:407] Connection header received
0927 14:51:31.296955 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:51:31.296958 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:51:31.297000 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:51:31.297107 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:51:31.297511 (+ 404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:51:31.298082 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:51:31.298984 (+ 902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:51:31.299196 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:51:31.299296 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:51:31.300258 (+ 962us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:51:31.300262 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:51:31.300265 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 14:51:31.300287 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:51:31.301754 (+ 1467us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:51:31.302235 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:51:31.302238 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:51:31.302239 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:51:31.302317 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:51:31.302627 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:51:31.302632 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:51:31.302634 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:51:31.302798 (+ 164us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:51:31.302902 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0927 14:51:31.303166 (+ 264us) server_negotiation.cc:299] Negotiation successful
0927 14:51:31.303271 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260927 14:51:31.303426 1716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:51:31.296439 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40621 (local address 127.0.98.1:35931)
0927 14:51:31.296749 (+ 310us) negotiation.cc:107] Waiting for socket to connect
0927 14:51:31.296760 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 14:51:31.296862 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:51:31.297128 (+ 266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:51:31.297131 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:51:31.297311 (+ 180us) client_negotiation.cc:189] Negotiated authn=SASL
0927 14:51:31.297398 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:51:31.297403 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:51:31.298201 (+ 798us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:51:31.298205 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:51:31.298860 (+ 655us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:51:31.298868 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:51:31.299300 (+ 432us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:51:31.299304 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:51:31.299395 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:51:31.299821 (+ 426us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 14:51:31.299831 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0927 14:51:31.300136 (+ 305us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 14:51:31.301866 (+ 1730us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 14:51:31.301868 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 14:51:31.301869 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 14:51:31.302125 (+ 256us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 14:51:31.302441 (+ 316us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 14:51:31.302444 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 14:51:31.302446 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 14:51:31.302536 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 14:51:31.302892 (+ 356us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 14:51:31.302895 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 14:51:31.303002 (+ 107us) client_negotiation.cc:769] Sending connection context
0927 14:51:31.303105 (+ 103us) client_negotiation.cc:240] Negotiation successful
0927 14:51:31.303243 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":243,"thread_start_us":87,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790520684602251-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790520684602251-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260927 14:51:31.328725 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6936m5/build/debug/bin/kudu
/tmp/dist-test-task6936m5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790520684602251-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790520684602251-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790520684602251-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790520684602251-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40621
--keytab_file=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790520684602251-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40231
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790520684602251-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790520684602251-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790520684602251-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 14:51:31.441522 1719 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 14:51:31.441859 1719 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 14:51:31.441964 1719 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 14:51:31.448177 1719 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 14:51:31.448282 1719 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 14:51:31.448374 1719 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 14:51:31.448462 1719 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 14:51:31.455706 1719 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40231
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790520684602251-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790520684602251-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790520684602251-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790520684602251-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:40621
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1719
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790520684602251-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:15:17 UTC on ccycloud.quasar-ledxfe.root.comops.site
build id 4676134
I20260927 14:51:31.456802 1719 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 14:51:31.458029 1719 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 14:51:31.467790 1724 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 14:51:31.467906 1719 server_base.cc:1034] running on GCE node
W20260927 14:51:31.467809 1727 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 14:51:31.467815 1725 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 14:51:31.468532 1719 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 14:51:31.469233 1719 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 14:51:31.470402 1719 hybrid_clock.cc:648] HybridClock initialized: now 1790520691470385 us; error 32 us; skew 500 ppm
Sep 27 14:51:31 dist-test-slave-xk1g krb5kdc[1462](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790520691, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 14:51:31.473227 1719 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 14:51:31.474109 1719 webserver.cc:460] Webserver started at http://127.0.98.2:39333/ using document root <none> and password file <none>
I20260927 14:51:31.474690 1719 fs_manager.cc:359] Metadata directory not provided
I20260927 14:51:31.474752 1719 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 14:51:31.474983 1719 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 14:51:31.477160 1719 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790520684602251-392-0/minicluster-data/ts-1/data/instance:
uuid: "124b623aa74049c7baeec32f6adcf3af"
format_stamp: "Formatted at 2026-09-27 14:51:31 on dist-test-slave-xk1g"
server_key: "d29bbc7d2b1c6576a1d3e1574b90a188"
server_key_iv: "b7212842f6f2f9572db01bf02b0e78f1"
server_key_version: "encryptionkey@0"
I20260927 14:51:31.477718 1719 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790520684602251-392-0/minicluster-data/ts-1/wal/instance:
uuid: "124b623aa74049c7baeec32f6adcf3af"
format_stamp: "Formatted at 2026-09-27 14:51:31 on dist-test-slave-xk1g"
server_key: "d29bbc7d2b1c6576a1d3e1574b90a188"
server_key_iv: "b7212842f6f2f9572db01bf02b0e78f1"
server_key_version: "encryptionkey@0"
I20260927 14:51:31.481415 1719 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260927 14:51:31.483790 1734 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 14:51:31.485200 1719 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 14:51:31.485320 1719 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790520684602251-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790520684602251-392-0/minicluster-data/ts-1/wal
uuid: "124b623aa74049c7baeec32f6adcf3af"
format_stamp: "Formatted at 2026-09-27 14:51:31 on dist-test-slave-xk1g"
server_key: "d29bbc7d2b1c6576a1d3e1574b90a188"
server_key_iv: "b7212842f6f2f9572db01bf02b0e78f1"
server_key_version: "encryptionkey@0"
I20260927 14:51:31.485425 1719 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790520684602251-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790520684602251-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790520684602251-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 14:51:31.511024 1719 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 14:51:31.514674 1719 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 14:51:31.514889 1719 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 14:51:31.516376 1719 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 14:51:31.516461 1719 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 14:51:31.516534 1719 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 14:51:31.516578 1719 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 14:51:31.527271 1719 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:44477
I20260927 14:51:31.527349 1849 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:44477 every 8 connection(s)
Sep 27 14:51:31 dist-test-slave-xk1g krb5kdc[1462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790520691, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 14:51:31.528347 1719 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790520684602251-392-0/minicluster-data/ts-1/data/info.pb
I20260927 14:51:31.532306 1556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:51:31.517388 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:32985 (local address 127.0.98.62:40621)
0927 14:51:31.517529 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0927 14:51:31.517532 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 14:51:31.518683 (+ 1151us) server_negotiation.cc:407] Connection header received
0927 14:51:31.519831 (+ 1148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:51:31.519835 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:51:31.519902 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:51:31.519997 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:51:31.521763 (+ 1766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:51:31.522619 (+ 856us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:51:31.524502 (+ 1883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:51:31.524714 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:51:31.524799 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:51:31.528176 (+ 3377us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:51:31.528179 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:51:31.528181 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 14:51:31.528208 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:51:31.529890 (+ 1682us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:51:31.530669 (+ 779us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:51:31.530672 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:51:31.530673 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:51:31.530755 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:51:31.531179 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:51:31.531182 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:51:31.531183 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:51:31.531383 (+ 200us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:51:31.531474 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0927 14:51:31.532073 (+ 599us) server_negotiation.cc:299] Negotiation successful
0927 14:51:31.532186 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260927 14:51:31.533231 1743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:51:31.518009 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40621 (local address 127.0.98.2:32985)
0927 14:51:31.518521 (+ 512us) negotiation.cc:107] Waiting for socket to connect
0927 14:51:31.518566 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0927 14:51:31.519617 (+ 1051us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:51:31.520238 (+ 621us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:51:31.520250 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:51:31.520742 (+ 492us) client_negotiation.cc:189] Negotiated authn=SASL
0927 14:51:31.521563 (+ 821us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:51:31.521587 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:51:31.522765 (+ 1178us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:51:31.522770 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:51:31.524377 (+ 1607us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:51:31.524387 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:51:31.524876 (+ 489us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:51:31.524880 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:51:31.525188 (+ 308us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:51:31.525963 (+ 775us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 14:51:31.525989 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0927 14:51:31.527955 (+ 1966us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 14:51:31.530089 (+ 2134us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 14:51:31.530094 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 14:51:31.530106 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0927 14:51:31.530554 (+ 448us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 14:51:31.530884 (+ 330us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 14:51:31.530887 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 14:51:31.530888 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 14:51:31.531055 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 14:51:31.531516 (+ 461us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 14:51:31.531522 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 14:51:31.531763 (+ 241us) client_negotiation.cc:769] Sending connection context
0927 14:51:31.531978 (+ 215us) client_negotiation.cc:240] Negotiation successful
0927 14:51:31.532249 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":306,"thread_start_us":143,"threads_started":1}
I20260927 14:51:31.534876 1850 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40621
I20260927 14:51:31.535068 1850 heartbeater.cc:464] Registering TS with master...
I20260927 14:51:31.535383 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6936m5/build/debug/bin/kudu as pid 1719
I20260927 14:51:31.535499 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790520684602251-392-0/minicluster-data/ts-1/wal/instance
I20260927 14:51:31.535624 1850 heartbeater.cc:511] Master 127.0.98.62:40621 requested a full tablet report, sending...
I20260927 14:51:31.535830 392 external_mini_cluster.cc:1428] Setting key f8b1965701364f5c8bf9cb7d61ba8ba2
I20260927 14:51:31.536716 1506 ts_manager.cc:194] Registered new tserver with Master: 124b623aa74049c7baeec32f6adcf3af (127.0.98.2:44477)
I20260927 14:51:31.537459 1506 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:32985
I20260927 14:51:31.544898 1556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:51:31.537196 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:51275 (local address 127.0.98.62:40621)
0927 14:51:31.537306 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0927 14:51:31.537310 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 14:51:31.537555 (+ 245us) server_negotiation.cc:407] Connection header received
0927 14:51:31.537673 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:51:31.537679 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:51:31.537733 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:51:31.537807 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:51:31.538339 (+ 532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:51:31.538936 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:51:31.540030 (+ 1094us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:51:31.540234 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:51:31.540313 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:51:31.541711 (+ 1398us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:51:31.541714 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:51:31.541716 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 14:51:31.541736 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:51:31.543210 (+ 1474us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:51:31.543745 (+ 535us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:51:31.543748 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:51:31.543749 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:51:31.543827 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:51:31.544121 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:51:31.544123 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:51:31.544124 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:51:31.544317 (+ 193us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:51:31.544393 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0927 14:51:31.544688 (+ 295us) server_negotiation.cc:299] Negotiation successful
0927 14:51:31.544776 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260927 14:51:31.545037 1856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:51:31.537139 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40621 (local address 127.0.98.2:51275)
0927 14:51:31.537473 (+ 334us) negotiation.cc:107] Waiting for socket to connect
0927 14:51:31.537485 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 14:51:31.537595 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:51:31.537924 (+ 329us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:51:31.537927 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:51:31.538131 (+ 204us) client_negotiation.cc:189] Negotiated authn=SASL
0927 14:51:31.538227 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:51:31.538232 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:51:31.539053 (+ 821us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:51:31.539056 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:51:31.539901 (+ 845us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:51:31.539913 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:51:31.540346 (+ 433us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:51:31.540353 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:51:31.540491 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:51:31.541043 (+ 552us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 14:51:31.541083 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0927 14:51:31.541585 (+ 502us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 14:51:31.543317 (+ 1732us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 14:51:31.543320 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 14:51:31.543323 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 14:51:31.543610 (+ 287us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 14:51:31.543925 (+ 315us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 14:51:31.543929 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 14:51:31.543933 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 14:51:31.544034 (+ 101us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 14:51:31.544410 (+ 376us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 14:51:31.544416 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 14:51:31.544531 (+ 115us) client_negotiation.cc:769] Sending connection context
0927 14:51:31.544650 (+ 119us) client_negotiation.cc:240] Negotiation successful
0927 14:51:31.544805 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":246,"thread_start_us":104,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790520684602251-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790520684602251-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260927 14:51:31.564144 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6936m5/build/debug/bin/kudu
/tmp/dist-test-task6936m5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790520684602251-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790520684602251-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790520684602251-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790520684602251-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40621
--keytab_file=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790520684602251-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40231
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790520684602251-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790520684602251-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790520684602251-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 14:51:31.677460 1859 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 14:51:31.677739 1859 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 14:51:31.677784 1859 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 14:51:31.681921 1859 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 14:51:31.681989 1859 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 14:51:31.682030 1859 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 14:51:31.682065 1859 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 14:51:31.687413 1859 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40231
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790520684602251-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790520684602251-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790520684602251-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790520684602251-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:40621
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1859
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790520684602251-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:15:17 UTC on ccycloud.quasar-ledxfe.root.comops.site
build id 4676134
I20260927 14:51:31.688503 1859 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 14:51:31.689586 1859 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 14:51:31.699088 1865 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 14:51:31.699082 1864 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 14:51:31.699082 1867 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 14:51:31.699194 1859 server_base.cc:1034] running on GCE node
I20260927 14:51:31.699785 1859 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 14:51:31.700417 1859 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 14:51:31.701566 1859 hybrid_clock.cc:648] HybridClock initialized: now 1790520691701549 us; error 31 us; skew 500 ppm
Sep 27 14:51:31 dist-test-slave-xk1g krb5kdc[1462](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790520691, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 14:51:31.704290 1859 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 14:51:31.705286 1859 webserver.cc:460] Webserver started at http://127.0.98.3:33273/ using document root <none> and password file <none>
I20260927 14:51:31.705818 1859 fs_manager.cc:359] Metadata directory not provided
I20260927 14:51:31.705875 1859 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 14:51:31.706039 1859 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 14:51:31.708053 1859 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790520684602251-392-0/minicluster-data/ts-2/data/instance:
uuid: "060ee5e35c494ed4a668fcbf2660ff99"
format_stamp: "Formatted at 2026-09-27 14:51:31 on dist-test-slave-xk1g"
server_key: "8a1def7216e00829a3958168decb5021"
server_key_iv: "611bf53430222a206a8aa84c4906aba9"
server_key_version: "encryptionkey@0"
I20260927 14:51:31.708583 1859 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790520684602251-392-0/minicluster-data/ts-2/wal/instance:
uuid: "060ee5e35c494ed4a668fcbf2660ff99"
format_stamp: "Formatted at 2026-09-27 14:51:31 on dist-test-slave-xk1g"
server_key: "8a1def7216e00829a3958168decb5021"
server_key_iv: "611bf53430222a206a8aa84c4906aba9"
server_key_version: "encryptionkey@0"
I20260927 14:51:31.712610 1859 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260927 14:51:31.715178 1874 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 14:51:31.716459 1859 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 14:51:31.716584 1859 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790520684602251-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790520684602251-392-0/minicluster-data/ts-2/wal
uuid: "060ee5e35c494ed4a668fcbf2660ff99"
format_stamp: "Formatted at 2026-09-27 14:51:31 on dist-test-slave-xk1g"
server_key: "8a1def7216e00829a3958168decb5021"
server_key_iv: "611bf53430222a206a8aa84c4906aba9"
server_key_version: "encryptionkey@0"
I20260927 14:51:31.716697 1859 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790520684602251-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790520684602251-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790520684602251-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 14:51:31.742022 1859 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 14:51:31.744936 1859 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 14:51:31.745157 1859 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 14:51:31.746623 1859 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 14:51:31.746678 1859 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 14:51:31.746718 1859 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 14:51:31.746768 1859 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 27 14:51:31 dist-test-slave-xk1g krb5kdc[1462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790520691, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 14:51:31.757807 1859 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:45063
I20260927 14:51:31.757846 1989 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:45063 every 8 connection(s)
I20260927 14:51:31.758947 1859 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790520684602251-392-0/minicluster-data/ts-2/data/info.pb
I20260927 14:51:31.760546 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6936m5/build/debug/bin/kudu as pid 1859
I20260927 14:51:31.760680 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790520684602251-392-0/minicluster-data/ts-2/wal/instance
I20260927 14:51:31.760823 1556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:51:31.747481 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:39967 (local address 127.0.98.62:40621)
0927 14:51:31.747633 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0927 14:51:31.747638 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 14:51:31.748363 (+ 725us) server_negotiation.cc:407] Connection header received
0927 14:51:31.749407 (+ 1044us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:51:31.749413 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:51:31.749477 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:51:31.749572 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:51:31.750972 (+ 1400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:51:31.751541 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:51:31.753409 (+ 1868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:51:31.753702 (+ 293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:51:31.753821 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:51:31.756794 (+ 2973us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:51:31.756798 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:51:31.756800 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 14:51:31.756832 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:51:31.758679 (+ 1847us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:51:31.759181 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:51:31.759184 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:51:31.759186 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:51:31.759283 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:51:31.759719 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:51:31.759723 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:51:31.759725 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:51:31.759945 (+ 220us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:51:31.760031 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0927 14:51:31.760518 (+ 487us) server_negotiation.cc:299] Negotiation successful
0927 14:51:31.760657 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260927 14:51:31.761000 392 external_mini_cluster.cc:1428] Setting key a037c5583cca220389bfab42f4e17a0b
I20260927 14:51:31.761740 1883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:51:31.747762 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40621 (local address 127.0.98.3:39967)
0927 14:51:31.748200 (+ 438us) negotiation.cc:107] Waiting for socket to connect
0927 14:51:31.748250 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0927 14:51:31.749162 (+ 912us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:51:31.749855 (+ 693us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:51:31.749874 (+ 19us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:51:31.750283 (+ 409us) client_negotiation.cc:189] Negotiated authn=SASL
0927 14:51:31.750826 (+ 543us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:51:31.750839 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:51:31.751663 (+ 824us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:51:31.751667 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:51:31.753271 (+ 1604us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:51:31.753283 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:51:31.753868 (+ 585us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:51:31.753872 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:51:31.754044 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:51:31.754654 (+ 610us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 14:51:31.754673 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0927 14:51:31.756608 (+ 1935us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 14:51:31.758785 (+ 2177us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 14:51:31.758790 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 14:51:31.758801 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0927 14:51:31.759077 (+ 276us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 14:51:31.759405 (+ 328us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 14:51:31.759409 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 14:51:31.759411 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 14:51:31.759594 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 14:51:31.760051 (+ 457us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 14:51:31.760056 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 14:51:31.760281 (+ 225us) client_negotiation.cc:769] Sending connection context
0927 14:51:31.760434 (+ 153us) client_negotiation.cc:240] Negotiation successful
0927 14:51:31.760741 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":242,"thread_start_us":133,"threads_started":1}
I20260927 14:51:31.762869 1990 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40621
I20260927 14:51:31.763032 1990 heartbeater.cc:464] Registering TS with master...
I20260927 14:51:31.763554 1990 heartbeater.cc:511] Master 127.0.98.62:40621 requested a full tablet report, sending...
I20260927 14:51:31.764544 1507 ts_manager.cc:194] Registered new tserver with Master: 060ee5e35c494ed4a668fcbf2660ff99 (127.0.98.3:45063)
I20260927 14:51:31.765300 1507 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:39967
I20260927 14:51:31.771767 1556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:51:31.765140 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:49109 (local address 127.0.98.62:40621)
0927 14:51:31.765260 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0927 14:51:31.765265 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 14:51:31.765442 (+ 177us) server_negotiation.cc:407] Connection header received
0927 14:51:31.765617 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:51:31.765621 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:51:31.765669 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:51:31.765715 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:51:31.766254 (+ 539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:51:31.766750 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:51:31.767558 (+ 808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:51:31.767786 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:51:31.767890 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:51:31.769017 (+ 1127us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:51:31.769020 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:51:31.769023 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 14:51:31.769046 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:51:31.770253 (+ 1207us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:51:31.770651 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:51:31.770655 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:51:31.770657 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:51:31.770735 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:51:31.771004 (+ 269us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:51:31.771007 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:51:31.771009 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:51:31.771173 (+ 164us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:51:31.771272 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0927 14:51:31.771510 (+ 238us) server_negotiation.cc:299] Negotiation successful
0927 14:51:31.771615 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260927 14:51:31.771783 1996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:51:31.765093 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40621 (local address 127.0.98.3:49109)
0927 14:51:31.765360 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0927 14:51:31.765377 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 14:51:31.765504 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:51:31.765848 (+ 344us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:51:31.765851 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:51:31.766040 (+ 189us) client_negotiation.cc:189] Negotiated authn=SASL
0927 14:51:31.766135 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:51:31.766139 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:51:31.766869 (+ 730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:51:31.766872 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:51:31.767448 (+ 576us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:51:31.767455 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:51:31.767891 (+ 436us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:51:31.767894 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:51:31.768005 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:51:31.768521 (+ 516us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 14:51:31.768532 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 14:51:31.768845 (+ 313us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 14:51:31.770355 (+ 1510us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 14:51:31.770360 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 14:51:31.770362 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 14:51:31.770556 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 14:51:31.770821 (+ 265us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 14:51:31.770825 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 14:51:31.770827 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 14:51:31.770912 (+ 85us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 14:51:31.771261 (+ 349us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 14:51:31.771264 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 14:51:31.771356 (+ 92us) client_negotiation.cc:769] Sending connection context
0927 14:51:31.771457 (+ 101us) client_negotiation.cc:240] Negotiation successful
0927 14:51:31.771600 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":175,"thread_start_us":100,"threads_started":1}
I20260927 14:51:31.774901 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 27 14:51:31 dist-test-slave-xk1g krb5kdc[1462](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790520691, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 27 14:51:31 dist-test-slave-xk1g krb5kdc[1462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790520691, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 14:51:31.799342 1556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:51:31.791466 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53370 (local address 127.0.98.62:40621)
0927 14:51:31.791586 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0927 14:51:31.791589 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 14:51:31.791600 (+ 11us) server_negotiation.cc:407] Connection header received
0927 14:51:31.791638 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:51:31.791640 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:51:31.791684 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:51:31.791773 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:51:31.792517 (+ 744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:51:31.793031 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:51:31.793898 (+ 867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:51:31.794116 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:51:31.794219 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:51:31.796500 (+ 2281us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:51:31.796504 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:51:31.796506 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 14:51:31.796529 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:51:31.797777 (+ 1248us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:51:31.798165 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:51:31.798171 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:51:31.798174 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:51:31.798248 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:51:31.798526 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:51:31.798529 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:51:31.798531 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:51:31.798701 (+ 170us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:51:31.798831 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0927 14:51:31.799064 (+ 233us) server_negotiation.cc:299] Negotiation successful
0927 14:51:31.799182 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260927 14:51:31.802073 1507 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:53370:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 14:51:31.805135 1507 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 14:51:31.819682 2012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:51:31.814224 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53292 (local address 127.0.98.1:43271)
0927 14:51:31.814644 (+ 420us) server_negotiation.cc:206] Beginning negotiation
0927 14:51:31.814652 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 14:51:31.814889 (+ 237us) server_negotiation.cc:407] Connection header received
0927 14:51:31.815034 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:51:31.815040 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:51:31.815205 (+ 165us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:51:31.815353 (+ 148us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 14:51:31.815728 (+ 375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:51:31.816830 (+ 1102us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:51:31.818502 (+ 1672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:51:31.819035 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:51:31.819162 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:51:31.819311 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0927 14:51:31.819444 (+ 133us) server_negotiation.cc:299] Negotiation successful
0927 14:51:31.819544 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":323,"thread_start_us":98,"threads_started":1}
I20260927 14:51:31.819682 2007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:51:31.814043 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:43271 (local address 127.0.0.1:53292)
0927 14:51:31.814790 (+ 747us) negotiation.cc:107] Waiting for socket to connect
0927 14:51:31.814802 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 14:51:31.814916 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:51:31.815342 (+ 426us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:51:31.815346 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:51:31.815363 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 14:51:31.815600 (+ 237us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:51:31.815608 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:51:31.816965 (+ 1357us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:51:31.816968 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:51:31.818347 (+ 1379us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:51:31.818357 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:51:31.819158 (+ 801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:51:31.819161 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:51:31.819264 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:51:31.819300 (+ 36us) client_negotiation.cc:769] Sending connection context
0927 14:51:31.819415 (+ 115us) client_negotiation.cc:240] Negotiation successful
0927 14:51:31.819539 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":697,"mutex_wait_us":181,"thread_start_us":81,"threads_started":1}
I20260927 14:51:31.820179 2008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:51:31.813303 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:45063 (local address 127.0.0.1:57502)
0927 14:51:31.814223 (+ 920us) negotiation.cc:107] Waiting for socket to connect
0927 14:51:31.814273 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0927 14:51:31.814400 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:51:31.815177 (+ 777us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:51:31.815183 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:51:31.815218 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 14:51:31.815504 (+ 286us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:51:31.815510 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:51:31.816580 (+ 1070us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:51:31.816587 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:51:31.818133 (+ 1546us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:51:31.818143 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:51:31.819464 (+ 1321us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:51:31.819467 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:51:31.819580 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:51:31.819612 (+ 32us) client_negotiation.cc:769] Sending connection context
0927 14:51:31.819690 (+ 78us) client_negotiation.cc:240] Negotiation successful
0927 14:51:31.819763 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":737,"mutex_wait_us":143,"spinlock_wait_cycles":6144,"thread_start_us":89,"threads_started":1}
I20260927 14:51:31.820500 2011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:51:31.813352 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:44477 (local address 127.0.0.1:52340)
0927 14:51:31.814348 (+ 996us) negotiation.cc:107] Waiting for socket to connect
0927 14:51:31.814359 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 14:51:31.814431 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:51:31.815177 (+ 746us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:51:31.815183 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:51:31.815217 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 14:51:31.815482 (+ 265us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:51:31.815492 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:51:31.816973 (+ 1481us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:51:31.816976 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:51:31.818347 (+ 1371us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:51:31.818357 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:51:31.819680 (+ 1323us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:51:31.819686 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:51:31.819770 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:51:31.819797 (+ 27us) client_negotiation.cc:769] Sending connection context
0927 14:51:31.819869 (+ 72us) client_negotiation.cc:240] Negotiation successful
0927 14:51:31.819940 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":944,"mutex_wait_us":170,"spinlock_wait_cycles":5248,"thread_start_us":96,"threads_started":1}
I20260927 14:51:31.820580 2009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:51:31.813389 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57502 (local address 127.0.98.3:45063)
0927 14:51:31.814691 (+ 1302us) server_negotiation.cc:206] Beginning negotiation
0927 14:51:31.814701 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 14:51:31.814719 (+ 18us) server_negotiation.cc:407] Connection header received
0927 14:51:31.814779 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:51:31.814782 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:51:31.814936 (+ 154us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:51:31.815052 (+ 116us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 14:51:31.815614 (+ 562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:51:31.816453 (+ 839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:51:31.818266 (+ 1813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:51:31.819114 (+ 848us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:51:31.819230 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:51:31.819396 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0927 14:51:31.819848 (+ 452us) server_negotiation.cc:299] Negotiation successful
0927 14:51:31.820422 (+ 574us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1210,"thread_start_us":106,"threads_started":1}
I20260927 14:51:31.820307 2010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:51:31.813570 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52340 (local address 127.0.98.2:44477)
0927 14:51:31.813961 (+ 391us) server_negotiation.cc:206] Beginning negotiation
0927 14:51:31.813975 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 14:51:31.814544 (+ 569us) server_negotiation.cc:407] Connection header received
0927 14:51:31.814626 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:51:31.814632 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:51:31.814838 (+ 206us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:51:31.815003 (+ 165us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 14:51:31.815629 (+ 626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:51:31.816830 (+ 1201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:51:31.818474 (+ 1644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:51:31.819347 (+ 873us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:51:31.819459 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:51:31.819615 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0927 14:51:31.820032 (+ 417us) server_negotiation.cc:299] Negotiation successful
0927 14:51:31.820139 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":273,"thread_start_us":90,"threads_started":1}
I20260927 14:51:31.823001 1924 tablet_service.cc:1467] Processing CreateTablet for tablet cd56edd40e394e35bc190148e2f91797 (DEFAULT_TABLE table=test-table [id=efce94dc76494038b8dedb338792cd63]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 14:51:31.823011 1644 tablet_service.cc:1467] Processing CreateTablet for tablet cd56edd40e394e35bc190148e2f91797 (DEFAULT_TABLE table=test-table [id=efce94dc76494038b8dedb338792cd63]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 14:51:31.823961 1924 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cd56edd40e394e35bc190148e2f91797. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 14:51:31.823954 1644 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cd56edd40e394e35bc190148e2f91797. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 14:51:31.827039 1782 tablet_service.cc:1467] Processing CreateTablet for tablet cd56edd40e394e35bc190148e2f91797 (DEFAULT_TABLE table=test-table [id=efce94dc76494038b8dedb338792cd63]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 14:51:31.828112 1782 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cd56edd40e394e35bc190148e2f91797. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 14:51:31.830297 2013 tablet_bootstrap.cc:492] T cd56edd40e394e35bc190148e2f91797 P 9c32f7b24fbc4b148527a0b1fedf8db8: Bootstrap starting.
I20260927 14:51:31.832496 2013 tablet_bootstrap.cc:654] T cd56edd40e394e35bc190148e2f91797 P 9c32f7b24fbc4b148527a0b1fedf8db8: Neither blocks nor log segments found. Creating new log.
I20260927 14:51:31.833313 2013 log.cc:824] T cd56edd40e394e35bc190148e2f91797 P 9c32f7b24fbc4b148527a0b1fedf8db8: Log is configured to *not* fsync() on all Append() calls
I20260927 14:51:31.833972 2014 tablet_bootstrap.cc:492] T cd56edd40e394e35bc190148e2f91797 P 060ee5e35c494ed4a668fcbf2660ff99: Bootstrap starting.
I20260927 14:51:31.835253 2013 tablet_bootstrap.cc:492] T cd56edd40e394e35bc190148e2f91797 P 9c32f7b24fbc4b148527a0b1fedf8db8: No bootstrap required, opened a new log
I20260927 14:51:31.835443 2013 ts_tablet_manager.cc:1397] T cd56edd40e394e35bc190148e2f91797 P 9c32f7b24fbc4b148527a0b1fedf8db8: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260927 14:51:31.835875 2016 tablet_bootstrap.cc:492] T cd56edd40e394e35bc190148e2f91797 P 124b623aa74049c7baeec32f6adcf3af: Bootstrap starting.
I20260927 14:51:31.836359 2014 tablet_bootstrap.cc:654] T cd56edd40e394e35bc190148e2f91797 P 060ee5e35c494ed4a668fcbf2660ff99: Neither blocks nor log segments found. Creating new log.
I20260927 14:51:31.837289 2014 log.cc:824] T cd56edd40e394e35bc190148e2f91797 P 060ee5e35c494ed4a668fcbf2660ff99: Log is configured to *not* fsync() on all Append() calls
I20260927 14:51:31.838153 2016 tablet_bootstrap.cc:654] T cd56edd40e394e35bc190148e2f91797 P 124b623aa74049c7baeec32f6adcf3af: Neither blocks nor log segments found. Creating new log.
I20260927 14:51:31.838975 2016 log.cc:824] T cd56edd40e394e35bc190148e2f91797 P 124b623aa74049c7baeec32f6adcf3af: Log is configured to *not* fsync() on all Append() calls
I20260927 14:51:31.839459 2014 tablet_bootstrap.cc:492] T cd56edd40e394e35bc190148e2f91797 P 060ee5e35c494ed4a668fcbf2660ff99: No bootstrap required, opened a new log
I20260927 14:51:31.839406 2013 raft_consensus.cc:348] T cd56edd40e394e35bc190148e2f91797 P 9c32f7b24fbc4b148527a0b1fedf8db8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "060ee5e35c494ed4a668fcbf2660ff99" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45063 } } peers { permanent_uuid: "124b623aa74049c7baeec32f6adcf3af" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44477 } } peers { permanent_uuid: "9c32f7b24fbc4b148527a0b1fedf8db8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43271 } }
I20260927 14:51:31.839690 2013 raft_consensus.cc:374] T cd56edd40e394e35bc190148e2f91797 P 9c32f7b24fbc4b148527a0b1fedf8db8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 14:51:31.839699 2014 ts_tablet_manager.cc:1397] T cd56edd40e394e35bc190148e2f91797 P 060ee5e35c494ed4a668fcbf2660ff99: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.003s
I20260927 14:51:31.839775 2013 raft_consensus.cc:729] T cd56edd40e394e35bc190148e2f91797 P 9c32f7b24fbc4b148527a0b1fedf8db8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9c32f7b24fbc4b148527a0b1fedf8db8, State: Initialized, Role: FOLLOWER
I20260927 14:51:31.840199 2013 consensus_queue.cc:260] T cd56edd40e394e35bc190148e2f91797 P 9c32f7b24fbc4b148527a0b1fedf8db8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "060ee5e35c494ed4a668fcbf2660ff99" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45063 } } peers { permanent_uuid: "124b623aa74049c7baeec32f6adcf3af" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44477 } } peers { permanent_uuid: "9c32f7b24fbc4b148527a0b1fedf8db8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43271 } }
I20260927 14:51:31.840758 1710 heartbeater.cc:503] Master 127.0.98.62:40621 was elected leader, sending a full tablet report...
I20260927 14:51:31.841123 2013 ts_tablet_manager.cc:1428] T cd56edd40e394e35bc190148e2f91797 P 9c32f7b24fbc4b148527a0b1fedf8db8: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260927 14:51:31.841132 2016 tablet_bootstrap.cc:492] T cd56edd40e394e35bc190148e2f91797 P 124b623aa74049c7baeec32f6adcf3af: No bootstrap required, opened a new log
I20260927 14:51:31.841490 2016 ts_tablet_manager.cc:1397] T cd56edd40e394e35bc190148e2f91797 P 124b623aa74049c7baeec32f6adcf3af: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260927 14:51:31.843935 2014 raft_consensus.cc:348] T cd56edd40e394e35bc190148e2f91797 P 060ee5e35c494ed4a668fcbf2660ff99 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "060ee5e35c494ed4a668fcbf2660ff99" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45063 } } peers { permanent_uuid: "124b623aa74049c7baeec32f6adcf3af" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44477 } } peers { permanent_uuid: "9c32f7b24fbc4b148527a0b1fedf8db8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43271 } }
I20260927 14:51:31.844187 2014 raft_consensus.cc:374] T cd56edd40e394e35bc190148e2f91797 P 060ee5e35c494ed4a668fcbf2660ff99 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 14:51:31.844250 2014 raft_consensus.cc:729] T cd56edd40e394e35bc190148e2f91797 P 060ee5e35c494ed4a668fcbf2660ff99 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 060ee5e35c494ed4a668fcbf2660ff99, State: Initialized, Role: FOLLOWER
I20260927 14:51:31.844719 2014 consensus_queue.cc:260] T cd56edd40e394e35bc190148e2f91797 P 060ee5e35c494ed4a668fcbf2660ff99 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "060ee5e35c494ed4a668fcbf2660ff99" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45063 } } peers { permanent_uuid: "124b623aa74049c7baeec32f6adcf3af" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44477 } } peers { permanent_uuid: "9c32f7b24fbc4b148527a0b1fedf8db8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43271 } }
I20260927 14:51:31.845393 1990 heartbeater.cc:503] Master 127.0.98.62:40621 was elected leader, sending a full tablet report...
I20260927 14:51:31.845801 2014 ts_tablet_manager.cc:1428] T cd56edd40e394e35bc190148e2f91797 P 060ee5e35c494ed4a668fcbf2660ff99: Time spent starting tablet: real 0.006s user 0.004s sys 0.003s
I20260927 14:51:31.846987 2016 raft_consensus.cc:348] T cd56edd40e394e35bc190148e2f91797 P 124b623aa74049c7baeec32f6adcf3af [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "060ee5e35c494ed4a668fcbf2660ff99" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45063 } } peers { permanent_uuid: "124b623aa74049c7baeec32f6adcf3af" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44477 } } peers { permanent_uuid: "9c32f7b24fbc4b148527a0b1fedf8db8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43271 } }
I20260927 14:51:31.847244 2016 raft_consensus.cc:374] T cd56edd40e394e35bc190148e2f91797 P 124b623aa74049c7baeec32f6adcf3af [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 14:51:31.847321 2016 raft_consensus.cc:729] T cd56edd40e394e35bc190148e2f91797 P 124b623aa74049c7baeec32f6adcf3af [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 124b623aa74049c7baeec32f6adcf3af, State: Initialized, Role: FOLLOWER
I20260927 14:51:31.847723 2016 consensus_queue.cc:260] T cd56edd40e394e35bc190148e2f91797 P 124b623aa74049c7baeec32f6adcf3af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "060ee5e35c494ed4a668fcbf2660ff99" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45063 } } peers { permanent_uuid: "124b623aa74049c7baeec32f6adcf3af" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44477 } } peers { permanent_uuid: "9c32f7b24fbc4b148527a0b1fedf8db8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43271 } }
I20260927 14:51:31.848408 1850 heartbeater.cc:503] Master 127.0.98.62:40621 was elected leader, sending a full tablet report...
I20260927 14:51:31.848647 2016 ts_tablet_manager.cc:1428] T cd56edd40e394e35bc190148e2f91797 P 124b623aa74049c7baeec32f6adcf3af: Time spent starting tablet: real 0.007s user 0.005s sys 0.000s
I20260927 14:51:31.964352 2021 raft_consensus.cc:482] T cd56edd40e394e35bc190148e2f91797 P 124b623aa74049c7baeec32f6adcf3af [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 14:51:31.964612 2021 raft_consensus.cc:504] T cd56edd40e394e35bc190148e2f91797 P 124b623aa74049c7baeec32f6adcf3af [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "060ee5e35c494ed4a668fcbf2660ff99" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45063 } } peers { permanent_uuid: "124b623aa74049c7baeec32f6adcf3af" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44477 } } peers { permanent_uuid: "9c32f7b24fbc4b148527a0b1fedf8db8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43271 } }
I20260927 14:51:31.965972 2021 leader_election.cc:288] T cd56edd40e394e35bc190148e2f91797 P 124b623aa74049c7baeec32f6adcf3af [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 060ee5e35c494ed4a668fcbf2660ff99 (127.0.98.3:45063), 9c32f7b24fbc4b148527a0b1fedf8db8 (127.0.98.1:43271)
I20260927 14:51:31.970731 1743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:51:31.966187 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:45063 (local address 127.0.98.2:38291)
0927 14:51:31.966323 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0927 14:51:31.966339 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 14:51:31.966434 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:51:31.966744 (+ 310us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:51:31.966748 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:51:31.966771 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 14:51:31.966911 (+ 140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:51:31.966916 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:51:31.968058 (+ 1142us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:51:31.968062 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:51:31.969453 (+ 1391us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:51:31.969465 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:51:31.970306 (+ 841us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:51:31.970309 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:51:31.970393 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:51:31.970413 (+ 20us) client_negotiation.cc:769] Sending connection context
0927 14:51:31.970481 (+ 68us) client_negotiation.cc:240] Negotiation successful
0927 14:51:31.970572 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":51}
I20260927 14:51:31.970752 2009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:51:31.966295 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:38291 (local address 127.0.98.3:45063)
0927 14:51:31.966480 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0927 14:51:31.966486 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 14:51:31.966504 (+ 18us) server_negotiation.cc:407] Connection header received
0927 14:51:31.966563 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:51:31.966566 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:51:31.966619 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:51:31.966695 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 14:51:31.967056 (+ 361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:51:31.967877 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:51:31.969602 (+ 1725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:51:31.970167 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:51:31.970284 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:51:31.970351 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0927 14:51:31.970528 (+ 177us) server_negotiation.cc:299] Negotiation successful
0927 14:51:31.970612 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260927 14:51:31.971037 2022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:51:31.966199 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:43271 (local address 127.0.98.2:35065)
0927 14:51:31.966688 (+ 489us) negotiation.cc:107] Waiting for socket to connect
0927 14:51:31.966704 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 14:51:31.966790 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:51:31.966994 (+ 204us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:51:31.966999 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:51:31.967019 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 14:51:31.967115 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:51:31.967119 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:51:31.968058 (+ 939us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:51:31.968062 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:51:31.969745 (+ 1683us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:51:31.969753 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:51:31.970612 (+ 859us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:51:31.970616 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:51:31.970707 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:51:31.970733 (+ 26us) client_negotiation.cc:769] Sending connection context
0927 14:51:31.970810 (+ 77us) client_negotiation.cc:240] Negotiation successful
0927 14:51:31.970879 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":420,"mutex_wait_us":113,"thread_start_us":164,"threads_started":1}
I20260927 14:51:31.971060 2012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:51:31.966308 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:35065 (local address 127.0.98.1:43271)
0927 14:51:31.966466 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0927 14:51:31.966473 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 14:51:31.966795 (+ 322us) server_negotiation.cc:407] Connection header received
0927 14:51:31.966855 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:51:31.966857 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:51:31.966897 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:51:31.966957 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 14:51:31.967216 (+ 259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:51:31.967947 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:51:31.969857 (+ 1910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:51:31.970484 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:51:31.970591 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:51:31.970647 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0927 14:51:31.970845 (+ 198us) server_negotiation.cc:299] Negotiation successful
0927 14:51:31.970929 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260927 14:51:31.971560 1944 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd56edd40e394e35bc190148e2f91797" candidate_uuid: "124b623aa74049c7baeec32f6adcf3af" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "060ee5e35c494ed4a668fcbf2660ff99" is_pre_election: true
I20260927 14:51:31.971760 1654 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd56edd40e394e35bc190148e2f91797" candidate_uuid: "124b623aa74049c7baeec32f6adcf3af" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c32f7b24fbc4b148527a0b1fedf8db8" is_pre_election: true
I20260927 14:51:31.971908 1944 raft_consensus.cc:2454] T cd56edd40e394e35bc190148e2f91797 P 060ee5e35c494ed4a668fcbf2660ff99 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 124b623aa74049c7baeec32f6adcf3af in term 0.
I20260927 14:51:31.972028 1654 raft_consensus.cc:2454] T cd56edd40e394e35bc190148e2f91797 P 9c32f7b24fbc4b148527a0b1fedf8db8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 124b623aa74049c7baeec32f6adcf3af in term 0.
I20260927 14:51:31.972503 1736 leader_election.cc:302] T cd56edd40e394e35bc190148e2f91797 P 124b623aa74049c7baeec32f6adcf3af [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 060ee5e35c494ed4a668fcbf2660ff99, 124b623aa74049c7baeec32f6adcf3af, 9c32f7b24fbc4b148527a0b1fedf8db8; no voters:
I20260927 14:51:31.972805 2021 raft_consensus.cc:2781] T cd56edd40e394e35bc190148e2f91797 P 124b623aa74049c7baeec32f6adcf3af [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 14:51:31.972889 2021 raft_consensus.cc:482] T cd56edd40e394e35bc190148e2f91797 P 124b623aa74049c7baeec32f6adcf3af [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 14:51:31.972939 2021 raft_consensus.cc:3037] T cd56edd40e394e35bc190148e2f91797 P 124b623aa74049c7baeec32f6adcf3af [term 0 FOLLOWER]: Advancing to term 1
I20260927 14:51:31.974076 2021 raft_consensus.cc:504] T cd56edd40e394e35bc190148e2f91797 P 124b623aa74049c7baeec32f6adcf3af [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "060ee5e35c494ed4a668fcbf2660ff99" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45063 } } peers { permanent_uuid: "124b623aa74049c7baeec32f6adcf3af" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44477 } } peers { permanent_uuid: "9c32f7b24fbc4b148527a0b1fedf8db8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43271 } }
I20260927 14:51:31.974419 2021 leader_election.cc:288] T cd56edd40e394e35bc190148e2f91797 P 124b623aa74049c7baeec32f6adcf3af [CANDIDATE]: Term 1 election: Requested vote from peers 060ee5e35c494ed4a668fcbf2660ff99 (127.0.98.3:45063), 9c32f7b24fbc4b148527a0b1fedf8db8 (127.0.98.1:43271)
I20260927 14:51:31.974766 1654 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd56edd40e394e35bc190148e2f91797" candidate_uuid: "124b623aa74049c7baeec32f6adcf3af" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c32f7b24fbc4b148527a0b1fedf8db8"
I20260927 14:51:31.974737 1944 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd56edd40e394e35bc190148e2f91797" candidate_uuid: "124b623aa74049c7baeec32f6adcf3af" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "060ee5e35c494ed4a668fcbf2660ff99"
I20260927 14:51:31.974890 1944 raft_consensus.cc:3037] T cd56edd40e394e35bc190148e2f91797 P 060ee5e35c494ed4a668fcbf2660ff99 [term 0 FOLLOWER]: Advancing to term 1
I20260927 14:51:31.974895 1654 raft_consensus.cc:3037] T cd56edd40e394e35bc190148e2f91797 P 9c32f7b24fbc4b148527a0b1fedf8db8 [term 0 FOLLOWER]: Advancing to term 1
I20260927 14:51:31.975934 1654 raft_consensus.cc:2454] T cd56edd40e394e35bc190148e2f91797 P 9c32f7b24fbc4b148527a0b1fedf8db8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 124b623aa74049c7baeec32f6adcf3af in term 1.
I20260927 14:51:31.976078 1944 raft_consensus.cc:2454] T cd56edd40e394e35bc190148e2f91797 P 060ee5e35c494ed4a668fcbf2660ff99 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 124b623aa74049c7baeec32f6adcf3af in term 1.
I20260927 14:51:31.976269 1736 leader_election.cc:302] T cd56edd40e394e35bc190148e2f91797 P 124b623aa74049c7baeec32f6adcf3af [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 124b623aa74049c7baeec32f6adcf3af, 9c32f7b24fbc4b148527a0b1fedf8db8; no voters:
I20260927 14:51:31.976438 2021 raft_consensus.cc:2781] T cd56edd40e394e35bc190148e2f91797 P 124b623aa74049c7baeec32f6adcf3af [term 1 FOLLOWER]: Leader election won for term 1
I20260927 14:51:31.976642 2021 raft_consensus.cc:686] T cd56edd40e394e35bc190148e2f91797 P 124b623aa74049c7baeec32f6adcf3af [term 1 LEADER]: Becoming Leader. State: Replica: 124b623aa74049c7baeec32f6adcf3af, State: Running, Role: LEADER
I20260927 14:51:31.976964 2021 consensus_queue.cc:237] T cd56edd40e394e35bc190148e2f91797 P 124b623aa74049c7baeec32f6adcf3af [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "060ee5e35c494ed4a668fcbf2660ff99" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45063 } } peers { permanent_uuid: "124b623aa74049c7baeec32f6adcf3af" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44477 } } peers { permanent_uuid: "9c32f7b24fbc4b148527a0b1fedf8db8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43271 } }
I20260927 14:51:31.980129 1506 catalog_manager.cc:5495] T cd56edd40e394e35bc190148e2f91797 P 124b623aa74049c7baeec32f6adcf3af reported cstate change: term changed from 0 to 1, leader changed from <none> to 124b623aa74049c7baeec32f6adcf3af (127.0.98.2). New cstate: current_term: 1 leader_uuid: "124b623aa74049c7baeec32f6adcf3af" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "060ee5e35c494ed4a668fcbf2660ff99" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45063 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "124b623aa74049c7baeec32f6adcf3af" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44477 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9c32f7b24fbc4b148527a0b1fedf8db8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43271 } health_report { overall_health: UNKNOWN } } }
W20260927 14:51:32.010231 1991 tablet.cc:2369] T cd56edd40e394e35bc190148e2f91797 P 060ee5e35c494ed4a668fcbf2660ff99: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 14:51:32.016762 2010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:51:32.012759 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52342 (local address 127.0.98.2:44477)
0927 14:51:32.012913 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0927 14:51:32.012918 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 14:51:32.012934 (+ 16us) server_negotiation.cc:407] Connection header received
0927 14:51:32.013225 (+ 291us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:51:32.013227 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:51:32.013288 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:51:32.013381 (+ 93us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 14:51:32.013672 (+ 291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:51:32.014203 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:51:32.015123 (+ 920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:51:32.015351 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:51:32.015436 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:51:32.015760 (+ 324us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 14:51:32.016224 (+ 464us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 14:51:32.016313 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0927 14:51:32.016483 (+ 170us) server_negotiation.cc:299] Negotiation successful
0927 14:51:32.016596 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260927 14:51:32.022866 1944 raft_consensus.cc:1261] T cd56edd40e394e35bc190148e2f91797 P 060ee5e35c494ed4a668fcbf2660ff99 [term 1 FOLLOWER]: Refusing update from remote peer 124b623aa74049c7baeec32f6adcf3af: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 14:51:32.022864 1654 raft_consensus.cc:1261] T cd56edd40e394e35bc190148e2f91797 P 9c32f7b24fbc4b148527a0b1fedf8db8 [term 1 FOLLOWER]: Refusing update from remote peer 124b623aa74049c7baeec32f6adcf3af: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 14:51:32.023653 2023 consensus_queue.cc:1035] T cd56edd40e394e35bc190148e2f91797 P 124b623aa74049c7baeec32f6adcf3af [LEADER]: Connected to new peer: Peer: permanent_uuid: "060ee5e35c494ed4a668fcbf2660ff99" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45063 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 14:51:32.023880 2021 consensus_queue.cc:1035] T cd56edd40e394e35bc190148e2f91797 P 124b623aa74049c7baeec32f6adcf3af [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c32f7b24fbc4b148527a0b1fedf8db8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43271 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260927 14:51:32.029933 1851 tablet.cc:2369] T cd56edd40e394e35bc190148e2f91797 P 124b623aa74049c7baeec32f6adcf3af: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 14:51:32.032286 2026 mvcc.cc:205] Tried to move back new op lower bound from 7333972754517716992 to 7333972754340544512. Current Snapshot: MvccSnapshot[applied={T|T < 7333972754517716992}]
I20260927 14:51:32.046939 1506 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:53370:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 14:51:32.047181 1506 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:53370:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 14:51:32.049968 1506 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a03aa1549f69443084f0c6a8c1b06bd6: Sending DeleteTablet for 3 replicas of tablet cd56edd40e394e35bc190148e2f91797
I20260927 14:51:32.050885 1644 tablet_service.cc:1514] Processing DeleteTablet for tablet cd56edd40e394e35bc190148e2f91797 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 14:51:32 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:53292
I20260927 14:51:32.050936 1924 tablet_service.cc:1514] Processing DeleteTablet for tablet cd56edd40e394e35bc190148e2f91797 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 14:51:32 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:57502
I20260927 14:51:32.050904 1782 tablet_service.cc:1514] Processing DeleteTablet for tablet cd56edd40e394e35bc190148e2f91797 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 14:51:32 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:52340
I20260927 14:51:32.051458 2037 tablet_replica.cc:330] T cd56edd40e394e35bc190148e2f91797 P 9c32f7b24fbc4b148527a0b1fedf8db8: stopping tablet replica
I20260927 14:51:32.051807 2036 tablet_replica.cc:330] T cd56edd40e394e35bc190148e2f91797 P 124b623aa74049c7baeec32f6adcf3af: stopping tablet replica
I20260927 14:51:32.051864 2037 raft_consensus.cc:2229] T cd56edd40e394e35bc190148e2f91797 P 9c32f7b24fbc4b148527a0b1fedf8db8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 14:51:32.052136 2036 raft_consensus.cc:2229] T cd56edd40e394e35bc190148e2f91797 P 124b623aa74049c7baeec32f6adcf3af [term 1 LEADER]: Raft consensus shutting down.
I20260927 14:51:32.052227 2037 raft_consensus.cc:2258] T cd56edd40e394e35bc190148e2f91797 P 9c32f7b24fbc4b148527a0b1fedf8db8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 14:51:32.052598 2036 raft_consensus.cc:2258] T cd56edd40e394e35bc190148e2f91797 P 124b623aa74049c7baeec32f6adcf3af [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 14:51:32.052762 1506 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:53370:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 14:51:32.053488 1506 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 14:51:32.053922 2037 ts_tablet_manager.cc:1905] T cd56edd40e394e35bc190148e2f91797 P 9c32f7b24fbc4b148527a0b1fedf8db8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 14:51:32.054504 2036 ts_tablet_manager.cc:1905] T cd56edd40e394e35bc190148e2f91797 P 124b623aa74049c7baeec32f6adcf3af: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 14:51:32.056793 2038 tablet_replica.cc:330] T cd56edd40e394e35bc190148e2f91797 P 060ee5e35c494ed4a668fcbf2660ff99: stopping tablet replica
I20260927 14:51:32.056833 2037 ts_tablet_manager.cc:1918] T cd56edd40e394e35bc190148e2f91797 P 9c32f7b24fbc4b148527a0b1fedf8db8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 14:51:32.056934 2037 log.cc:1196] T cd56edd40e394e35bc190148e2f91797 P 9c32f7b24fbc4b148527a0b1fedf8db8: Deleting WAL directory at /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790520684602251-392-0/minicluster-data/ts-0/wal/wals/cd56edd40e394e35bc190148e2f91797
I20260927 14:51:32.057170 2038 raft_consensus.cc:2229] T cd56edd40e394e35bc190148e2f91797 P 060ee5e35c494ed4a668fcbf2660ff99 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 14:51:32.057327 2037 ts_tablet_manager.cc:1939] T cd56edd40e394e35bc190148e2f91797 P 9c32f7b24fbc4b148527a0b1fedf8db8: Deleting consensus metadata
I20260927 14:51:32.057521 2038 raft_consensus.cc:2258] T cd56edd40e394e35bc190148e2f91797 P 060ee5e35c494ed4a668fcbf2660ff99 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 14:51:32.059221 2038 ts_tablet_manager.cc:1905] T cd56edd40e394e35bc190148e2f91797 P 060ee5e35c494ed4a668fcbf2660ff99: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 14:51:32.059988 1924 tablet_service.cc:1467] Processing CreateTablet for tablet af6c9e9c0c85420e8252fcdb9383e699 (DEFAULT_TABLE table=test-table [id=a38a3e6c14c742718d93fed486a865ee]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 14:51:32.060283 1924 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet af6c9e9c0c85420e8252fcdb9383e699. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 14:51:32.060457 1492 catalog_manager.cc:4841] TS 9c32f7b24fbc4b148527a0b1fedf8db8 (127.0.98.1:43271): tablet cd56edd40e394e35bc190148e2f91797 (table test-table [id=efce94dc76494038b8dedb338792cd63]) successfully deleted
I20260927 14:51:32.060614 1782 tablet_service.cc:1467] Processing CreateTablet for tablet af6c9e9c0c85420e8252fcdb9383e699 (DEFAULT_TABLE table=test-table [id=a38a3e6c14c742718d93fed486a865ee]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 14:51:32.060909 1782 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet af6c9e9c0c85420e8252fcdb9383e699. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 14:51:32.061034 1644 tablet_service.cc:1467] Processing CreateTablet for tablet af6c9e9c0c85420e8252fcdb9383e699 (DEFAULT_TABLE table=test-table [id=a38a3e6c14c742718d93fed486a865ee]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 14:51:32.061309 1644 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet af6c9e9c0c85420e8252fcdb9383e699. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 14:51:32.062956 2014 tablet_bootstrap.cc:492] T af6c9e9c0c85420e8252fcdb9383e699 P 060ee5e35c494ed4a668fcbf2660ff99: Bootstrap starting.
I20260927 14:51:32.063584 2016 tablet_bootstrap.cc:492] T af6c9e9c0c85420e8252fcdb9383e699 P 124b623aa74049c7baeec32f6adcf3af: Bootstrap starting.
I20260927 14:51:32.064047 2014 tablet_bootstrap.cc:654] T af6c9e9c0c85420e8252fcdb9383e699 P 060ee5e35c494ed4a668fcbf2660ff99: Neither blocks nor log segments found. Creating new log.
I20260927 14:51:32.064567 2016 tablet_bootstrap.cc:654] T af6c9e9c0c85420e8252fcdb9383e699 P 124b623aa74049c7baeec32f6adcf3af: Neither blocks nor log segments found. Creating new log.
I20260927 14:51:32.064646 2013 tablet_bootstrap.cc:492] T af6c9e9c0c85420e8252fcdb9383e699 P 9c32f7b24fbc4b148527a0b1fedf8db8: Bootstrap starting.
I20260927 14:51:32.065269 2014 tablet_bootstrap.cc:492] T af6c9e9c0c85420e8252fcdb9383e699 P 060ee5e35c494ed4a668fcbf2660ff99: No bootstrap required, opened a new log
I20260927 14:51:32.065341 2014 ts_tablet_manager.cc:1397] T af6c9e9c0c85420e8252fcdb9383e699 P 060ee5e35c494ed4a668fcbf2660ff99: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260927 14:51:32.065651 2013 tablet_bootstrap.cc:654] T af6c9e9c0c85420e8252fcdb9383e699 P 9c32f7b24fbc4b148527a0b1fedf8db8: Neither blocks nor log segments found. Creating new log.
I20260927 14:51:32.065827 2014 raft_consensus.cc:348] T af6c9e9c0c85420e8252fcdb9383e699 P 060ee5e35c494ed4a668fcbf2660ff99 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c32f7b24fbc4b148527a0b1fedf8db8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43271 } } peers { permanent_uuid: "124b623aa74049c7baeec32f6adcf3af" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44477 } } peers { permanent_uuid: "060ee5e35c494ed4a668fcbf2660ff99" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45063 } }
I20260927 14:51:32.065938 2014 raft_consensus.cc:374] T af6c9e9c0c85420e8252fcdb9383e699 P 060ee5e35c494ed4a668fcbf2660ff99 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 14:51:32.065982 2014 raft_consensus.cc:729] T af6c9e9c0c85420e8252fcdb9383e699 P 060ee5e35c494ed4a668fcbf2660ff99 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 060ee5e35c494ed4a668fcbf2660ff99, State: Initialized, Role: FOLLOWER
I20260927 14:51:32.066115 2014 consensus_queue.cc:260] T af6c9e9c0c85420e8252fcdb9383e699 P 060ee5e35c494ed4a668fcbf2660ff99 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c32f7b24fbc4b148527a0b1fedf8db8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43271 } } peers { permanent_uuid: "124b623aa74049c7baeec32f6adcf3af" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44477 } } peers { permanent_uuid: "060ee5e35c494ed4a668fcbf2660ff99" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45063 } }
I20260927 14:51:32.066397 2014 ts_tablet_manager.cc:1428] T af6c9e9c0c85420e8252fcdb9383e699 P 060ee5e35c494ed4a668fcbf2660ff99: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260927 14:51:32.066685 2036 ts_tablet_manager.cc:1918] T cd56edd40e394e35bc190148e2f91797 P 124b623aa74049c7baeec32f6adcf3af: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 14:51:32.066794 2036 log.cc:1196] T cd56edd40e394e35bc190148e2f91797 P 124b623aa74049c7baeec32f6adcf3af: Deleting WAL directory at /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790520684602251-392-0/minicluster-data/ts-1/wal/wals/cd56edd40e394e35bc190148e2f91797
I20260927 14:51:32.067900 2013 tablet_bootstrap.cc:492] T af6c9e9c0c85420e8252fcdb9383e699 P 9c32f7b24fbc4b148527a0b1fedf8db8: No bootstrap required, opened a new log
I20260927 14:51:32.068006 2013 ts_tablet_manager.cc:1397] T af6c9e9c0c85420e8252fcdb9383e699 P 9c32f7b24fbc4b148527a0b1fedf8db8: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20260927 14:51:32.068442 2013 raft_consensus.cc:348] T af6c9e9c0c85420e8252fcdb9383e699 P 9c32f7b24fbc4b148527a0b1fedf8db8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c32f7b24fbc4b148527a0b1fedf8db8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43271 } } peers { permanent_uuid: "124b623aa74049c7baeec32f6adcf3af" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44477 } } peers { permanent_uuid: "060ee5e35c494ed4a668fcbf2660ff99" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45063 } }
I20260927 14:51:32.068528 2013 raft_consensus.cc:374] T af6c9e9c0c85420e8252fcdb9383e699 P 9c32f7b24fbc4b148527a0b1fedf8db8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 14:51:32.068548 2013 raft_consensus.cc:729] T af6c9e9c0c85420e8252fcdb9383e699 P 9c32f7b24fbc4b148527a0b1fedf8db8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9c32f7b24fbc4b148527a0b1fedf8db8, State: Initialized, Role: FOLLOWER
I20260927 14:51:32.068678 2013 consensus_queue.cc:260] T af6c9e9c0c85420e8252fcdb9383e699 P 9c32f7b24fbc4b148527a0b1fedf8db8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c32f7b24fbc4b148527a0b1fedf8db8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43271 } } peers { permanent_uuid: "124b623aa74049c7baeec32f6adcf3af" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44477 } } peers { permanent_uuid: "060ee5e35c494ed4a668fcbf2660ff99" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45063 } }
I20260927 14:51:32.068946 2013 ts_tablet_manager.cc:1428] T af6c9e9c0c85420e8252fcdb9383e699 P 9c32f7b24fbc4b148527a0b1fedf8db8: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260927 14:51:32.069339 2036 ts_tablet_manager.cc:1939] T cd56edd40e394e35bc190148e2f91797 P 124b623aa74049c7baeec32f6adcf3af: Deleting consensus metadata
I20260927 14:51:32.070465 2038 ts_tablet_manager.cc:1918] T cd56edd40e394e35bc190148e2f91797 P 060ee5e35c494ed4a668fcbf2660ff99: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 14:51:32.070979 2038 log.cc:1196] T cd56edd40e394e35bc190148e2f91797 P 060ee5e35c494ed4a668fcbf2660ff99: Deleting WAL directory at /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790520684602251-392-0/minicluster-data/ts-2/wal/wals/cd56edd40e394e35bc190148e2f91797
I20260927 14:51:32.070992 1491 catalog_manager.cc:4841] TS 124b623aa74049c7baeec32f6adcf3af (127.0.98.2:44477): tablet cd56edd40e394e35bc190148e2f91797 (table test-table [id=efce94dc76494038b8dedb338792cd63]) successfully deleted
I20260927 14:51:32.071441 2038 ts_tablet_manager.cc:1939] T cd56edd40e394e35bc190148e2f91797 P 060ee5e35c494ed4a668fcbf2660ff99: Deleting consensus metadata
I20260927 14:51:32.071638 2016 tablet_bootstrap.cc:492] T af6c9e9c0c85420e8252fcdb9383e699 P 124b623aa74049c7baeec32f6adcf3af: No bootstrap required, opened a new log
I20260927 14:51:32.071739 2016 ts_tablet_manager.cc:1397] T af6c9e9c0c85420e8252fcdb9383e699 P 124b623aa74049c7baeec32f6adcf3af: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.000s
I20260927 14:51:32.072237 2016 raft_consensus.cc:348] T af6c9e9c0c85420e8252fcdb9383e699 P 124b623aa74049c7baeec32f6adcf3af [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c32f7b24fbc4b148527a0b1fedf8db8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43271 } } peers { permanent_uuid: "124b623aa74049c7baeec32f6adcf3af" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44477 } } peers { permanent_uuid: "060ee5e35c494ed4a668fcbf2660ff99" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45063 } }
I20260927 14:51:32.072371 2016 raft_consensus.cc:374] T af6c9e9c0c85420e8252fcdb9383e699 P 124b623aa74049c7baeec32f6adcf3af [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 14:51:32.072408 2016 raft_consensus.cc:729] T af6c9e9c0c85420e8252fcdb9383e699 P 124b623aa74049c7baeec32f6adcf3af [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 124b623aa74049c7baeec32f6adcf3af, State: Initialized, Role: FOLLOWER
I20260927 14:51:32.072546 2016 consensus_queue.cc:260] T af6c9e9c0c85420e8252fcdb9383e699 P 124b623aa74049c7baeec32f6adcf3af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c32f7b24fbc4b148527a0b1fedf8db8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43271 } } peers { permanent_uuid: "124b623aa74049c7baeec32f6adcf3af" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44477 } } peers { permanent_uuid: "060ee5e35c494ed4a668fcbf2660ff99" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45063 } }
I20260927 14:51:32.072698 1494 catalog_manager.cc:4841] TS 060ee5e35c494ed4a668fcbf2660ff99 (127.0.98.3:45063): tablet cd56edd40e394e35bc190148e2f91797 (table test-table [id=efce94dc76494038b8dedb338792cd63]) successfully deleted
I20260927 14:51:32.072822 2016 ts_tablet_manager.cc:1428] T af6c9e9c0c85420e8252fcdb9383e699 P 124b623aa74049c7baeec32f6adcf3af: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260927 14:51:32.129591 2019 raft_consensus.cc:482] T af6c9e9c0c85420e8252fcdb9383e699 P 9c32f7b24fbc4b148527a0b1fedf8db8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 14:51:32.129771 2019 raft_consensus.cc:504] T af6c9e9c0c85420e8252fcdb9383e699 P 9c32f7b24fbc4b148527a0b1fedf8db8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c32f7b24fbc4b148527a0b1fedf8db8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43271 } } peers { permanent_uuid: "124b623aa74049c7baeec32f6adcf3af" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44477 } } peers { permanent_uuid: "060ee5e35c494ed4a668fcbf2660ff99" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45063 } }
I20260927 14:51:32.131140 2019 leader_election.cc:288] T af6c9e9c0c85420e8252fcdb9383e699 P 9c32f7b24fbc4b148527a0b1fedf8db8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 124b623aa74049c7baeec32f6adcf3af (127.0.98.2:44477), 060ee5e35c494ed4a668fcbf2660ff99 (127.0.98.3:45063)
I20260927 14:51:32.136221 2010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:51:32.131793 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:41153 (local address 127.0.98.2:44477)
0927 14:51:32.131929 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0927 14:51:32.131934 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 14:51:32.131953 (+ 19us) server_negotiation.cc:407] Connection header received
0927 14:51:32.132103 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:51:32.132106 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:51:32.132162 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:51:32.132268 (+ 106us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 14:51:32.132932 (+ 664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:51:32.133605 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:51:32.134906 (+ 1301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:51:32.135551 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:51:32.135670 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:51:32.135770 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0927 14:51:32.136000 (+ 230us) server_negotiation.cc:299] Negotiation successful
0927 14:51:32.136075 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260927 14:51:32.136368 2009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:51:32.131543 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:59623 (local address 127.0.98.3:45063)
0927 14:51:32.131684 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0927 14:51:32.131690 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 14:51:32.131930 (+ 240us) server_negotiation.cc:407] Connection header received
0927 14:51:32.132119 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:51:32.132121 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:51:32.132161 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:51:32.132200 (+ 39us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 14:51:32.132649 (+ 449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:51:32.133449 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:51:32.135103 (+ 1654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:51:32.135696 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:51:32.135791 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:51:32.135852 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0927 14:51:32.136175 (+ 323us) server_negotiation.cc:299] Negotiation successful
0927 14:51:32.136255 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260927 14:51:32.136250 2043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:51:32.131432 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:44477 (local address 127.0.98.1:41153)
0927 14:51:32.131825 (+ 393us) negotiation.cc:107] Waiting for socket to connect
0927 14:51:32.131841 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 14:51:32.131966 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:51:32.132296 (+ 330us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:51:32.132302 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:51:32.132326 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 14:51:32.132518 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:51:32.132525 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:51:32.133732 (+ 1207us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:51:32.133742 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:51:32.134782 (+ 1040us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:51:32.134792 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:51:32.135710 (+ 918us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:51:32.135715 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:51:32.135837 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:51:32.135869 (+ 32us) client_negotiation.cc:769] Sending connection context
0927 14:51:32.135969 (+ 100us) client_negotiation.cc:240] Negotiation successful
0927 14:51:32.136053 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":319,"mutex_wait_us":32,"spinlock_wait_cycles":24704,"thread_start_us":84,"threads_started":1}
I20260927 14:51:32.136745 2042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:51:32.131432 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:45063 (local address 127.0.98.1:59623)
0927 14:51:32.131849 (+ 417us) negotiation.cc:107] Waiting for socket to connect
0927 14:51:32.131862 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 14:51:32.131980 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:51:32.132346 (+ 366us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:51:32.132348 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:51:32.132382 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 14:51:32.132525 (+ 143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:51:32.132531 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:51:32.133605 (+ 1074us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:51:32.133609 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:51:32.134948 (+ 1339us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:51:32.134961 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:51:32.135904 (+ 943us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:51:32.135906 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:51:32.136024 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:51:32.136047 (+ 23us) client_negotiation.cc:769] Sending connection context
0927 14:51:32.136497 (+ 450us) client_negotiation.cc:240] Negotiation successful
0927 14:51:32.136577 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":365,"mutex_wait_us":198,"spinlock_wait_cycles":384,"thread_start_us":88,"threads_started":1}
I20260927 14:51:32.137189 1944 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af6c9e9c0c85420e8252fcdb9383e699" candidate_uuid: "9c32f7b24fbc4b148527a0b1fedf8db8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "060ee5e35c494ed4a668fcbf2660ff99" is_pre_election: true
I20260927 14:51:32.137194 1804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af6c9e9c0c85420e8252fcdb9383e699" candidate_uuid: "9c32f7b24fbc4b148527a0b1fedf8db8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "124b623aa74049c7baeec32f6adcf3af" is_pre_election: true
I20260927 14:51:32.137316 1944 raft_consensus.cc:2454] T af6c9e9c0c85420e8252fcdb9383e699 P 060ee5e35c494ed4a668fcbf2660ff99 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c32f7b24fbc4b148527a0b1fedf8db8 in term 0.
I20260927 14:51:32.137430 1804 raft_consensus.cc:2454] T af6c9e9c0c85420e8252fcdb9383e699 P 124b623aa74049c7baeec32f6adcf3af [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c32f7b24fbc4b148527a0b1fedf8db8 in term 0.
I20260927 14:51:32.137672 1598 leader_election.cc:302] T af6c9e9c0c85420e8252fcdb9383e699 P 9c32f7b24fbc4b148527a0b1fedf8db8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 060ee5e35c494ed4a668fcbf2660ff99, 9c32f7b24fbc4b148527a0b1fedf8db8; no voters:
I20260927 14:51:32.138038 2019 raft_consensus.cc:2781] T af6c9e9c0c85420e8252fcdb9383e699 P 9c32f7b24fbc4b148527a0b1fedf8db8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 14:51:32.138105 2019 raft_consensus.cc:482] T af6c9e9c0c85420e8252fcdb9383e699 P 9c32f7b24fbc4b148527a0b1fedf8db8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 14:51:32.138147 2019 raft_consensus.cc:3037] T af6c9e9c0c85420e8252fcdb9383e699 P 9c32f7b24fbc4b148527a0b1fedf8db8 [term 0 FOLLOWER]: Advancing to term 1
I20260927 14:51:32.139432 2019 raft_consensus.cc:504] T af6c9e9c0c85420e8252fcdb9383e699 P 9c32f7b24fbc4b148527a0b1fedf8db8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c32f7b24fbc4b148527a0b1fedf8db8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43271 } } peers { permanent_uuid: "124b623aa74049c7baeec32f6adcf3af" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44477 } } peers { permanent_uuid: "060ee5e35c494ed4a668fcbf2660ff99" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45063 } }
I20260927 14:51:32.139859 2019 leader_election.cc:288] T af6c9e9c0c85420e8252fcdb9383e699 P 9c32f7b24fbc4b148527a0b1fedf8db8 [CANDIDATE]: Term 1 election: Requested vote from peers 124b623aa74049c7baeec32f6adcf3af (127.0.98.2:44477), 060ee5e35c494ed4a668fcbf2660ff99 (127.0.98.3:45063)
I20260927 14:51:32.140270 1804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af6c9e9c0c85420e8252fcdb9383e699" candidate_uuid: "9c32f7b24fbc4b148527a0b1fedf8db8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "124b623aa74049c7baeec32f6adcf3af"
I20260927 14:51:32.140409 1804 raft_consensus.cc:3037] T af6c9e9c0c85420e8252fcdb9383e699 P 124b623aa74049c7baeec32f6adcf3af [term 0 FOLLOWER]: Advancing to term 1
I20260927 14:51:32.140359 1944 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af6c9e9c0c85420e8252fcdb9383e699" candidate_uuid: "9c32f7b24fbc4b148527a0b1fedf8db8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "060ee5e35c494ed4a668fcbf2660ff99"
I20260927 14:51:32.140503 1944 raft_consensus.cc:3037] T af6c9e9c0c85420e8252fcdb9383e699 P 060ee5e35c494ed4a668fcbf2660ff99 [term 0 FOLLOWER]: Advancing to term 1
I20260927 14:51:32.141513 1944 raft_consensus.cc:2454] T af6c9e9c0c85420e8252fcdb9383e699 P 060ee5e35c494ed4a668fcbf2660ff99 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c32f7b24fbc4b148527a0b1fedf8db8 in term 1.
I20260927 14:51:32.141831 1598 leader_election.cc:302] T af6c9e9c0c85420e8252fcdb9383e699 P 9c32f7b24fbc4b148527a0b1fedf8db8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 060ee5e35c494ed4a668fcbf2660ff99, 9c32f7b24fbc4b148527a0b1fedf8db8; no voters:
I20260927 14:51:32.142021 2019 raft_consensus.cc:2781] T af6c9e9c0c85420e8252fcdb9383e699 P 9c32f7b24fbc4b148527a0b1fedf8db8 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 14:51:32.142246 1804 raft_consensus.cc:2454] T af6c9e9c0c85420e8252fcdb9383e699 P 124b623aa74049c7baeec32f6adcf3af [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c32f7b24fbc4b148527a0b1fedf8db8 in term 1.
I20260927 14:51:32.142280 2019 raft_consensus.cc:686] T af6c9e9c0c85420e8252fcdb9383e699 P 9c32f7b24fbc4b148527a0b1fedf8db8 [term 1 LEADER]: Becoming Leader. State: Replica: 9c32f7b24fbc4b148527a0b1fedf8db8, State: Running, Role: LEADER
I20260927 14:51:32.142550 2019 consensus_queue.cc:237] T af6c9e9c0c85420e8252fcdb9383e699 P 9c32f7b24fbc4b148527a0b1fedf8db8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c32f7b24fbc4b148527a0b1fedf8db8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43271 } } peers { permanent_uuid: "124b623aa74049c7baeec32f6adcf3af" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44477 } } peers { permanent_uuid: "060ee5e35c494ed4a668fcbf2660ff99" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45063 } }
I20260927 14:51:32.144991 1507 catalog_manager.cc:5495] T af6c9e9c0c85420e8252fcdb9383e699 P 9c32f7b24fbc4b148527a0b1fedf8db8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9c32f7b24fbc4b148527a0b1fedf8db8 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "9c32f7b24fbc4b148527a0b1fedf8db8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c32f7b24fbc4b148527a0b1fedf8db8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43271 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "124b623aa74049c7baeec32f6adcf3af" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44477 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "060ee5e35c494ed4a668fcbf2660ff99" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45063 } health_report { overall_health: UNKNOWN } } }
Sep 27 14:51:32 dist-test-slave-xk1g krb5kdc[1462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790520690, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 14:51:32.189767 2051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:51:32.178877 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40621 (local address 127.0.0.1:53378)
0927 14:51:32.179277 (+ 400us) negotiation.cc:107] Waiting for socket to connect
0927 14:51:32.179297 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0927 14:51:32.179445 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:51:32.179775 (+ 330us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:51:32.179778 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:51:32.180357 (+ 579us) client_negotiation.cc:189] Negotiated authn=SASL
0927 14:51:32.180493 (+ 136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:51:32.180503 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:51:32.181374 (+ 871us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:51:32.181379 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:51:32.182261 (+ 882us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:51:32.182274 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:51:32.182755 (+ 481us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:51:32.182760 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:51:32.182858 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:51:32.183568 (+ 710us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 14:51:32.183587 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0927 14:51:32.185456 (+ 1869us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 14:51:32.187625 (+ 2169us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 14:51:32.187640 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 14:51:32.187648 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0927 14:51:32.188020 (+ 372us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 14:51:32.188361 (+ 341us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 14:51:32.188365 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 14:51:32.188367 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 14:51:32.188477 (+ 110us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 14:51:32.188950 (+ 473us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 14:51:32.188959 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 14:51:32.189154 (+ 195us) client_negotiation.cc:769] Sending connection context
0927 14:51:32.189320 (+ 166us) client_negotiation.cc:240] Negotiation successful
0927 14:51:32.189533 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":288,"thread_start_us":159,"threads_started":1}
I20260927 14:51:32.189779 1556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:51:32.179037 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53378 (local address 127.0.98.62:40621)
0927 14:51:32.179168 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0927 14:51:32.179173 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 14:51:32.179389 (+ 216us) server_negotiation.cc:407] Connection header received
0927 14:51:32.179579 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:51:32.179583 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:51:32.179628 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:51:32.179715 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:51:32.180641 (+ 926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:51:32.181257 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:51:32.182408 (+ 1151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:51:32.182624 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:51:32.182712 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:51:32.185668 (+ 2956us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:51:32.185671 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:51:32.185673 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 14:51:32.185696 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:51:32.187495 (+ 1799us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:51:32.188131 (+ 636us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:51:32.188134 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:51:32.188136 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:51:32.188261 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:51:32.188575 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:51:32.188578 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:51:32.188580 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:51:32.188837 (+ 257us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:51:32.188951 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0927 14:51:32.189374 (+ 423us) server_negotiation.cc:299] Negotiation successful
0927 14:51:32.189660 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":164,"server-negotiator.queue_time_us":58}
I20260927 14:51:32.194346 1507 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:53378:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260927 14:51:32.195308 1507 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 14:51:32.200702 1782 tablet_service.cc:1467] Processing CreateTablet for tablet 8b641035254348abafd39e71d62d7a70 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=7df70809049d447d86092c0319c25757]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 14:51:32.200702 1644 tablet_service.cc:1467] Processing CreateTablet for tablet 8b641035254348abafd39e71d62d7a70 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=7df70809049d447d86092c0319c25757]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 14:51:32.201148 1644 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8b641035254348abafd39e71d62d7a70. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 14:51:32.201146 1782 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8b641035254348abafd39e71d62d7a70. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 14:51:32.204005 2016 tablet_bootstrap.cc:492] T 8b641035254348abafd39e71d62d7a70 P 124b623aa74049c7baeec32f6adcf3af: Bootstrap starting.
I20260927 14:51:32.204099 2013 tablet_bootstrap.cc:492] T 8b641035254348abafd39e71d62d7a70 P 9c32f7b24fbc4b148527a0b1fedf8db8: Bootstrap starting.
I20260927 14:51:32.204384 1924 tablet_service.cc:1467] Processing CreateTablet for tablet 8b641035254348abafd39e71d62d7a70 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=7df70809049d447d86092c0319c25757]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 14:51:32.204753 1924 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8b641035254348abafd39e71d62d7a70. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 14:51:32.205098 2013 tablet_bootstrap.cc:654] T 8b641035254348abafd39e71d62d7a70 P 9c32f7b24fbc4b148527a0b1fedf8db8: Neither blocks nor log segments found. Creating new log.
I20260927 14:51:32.205587 2016 tablet_bootstrap.cc:654] T 8b641035254348abafd39e71d62d7a70 P 124b623aa74049c7baeec32f6adcf3af: Neither blocks nor log segments found. Creating new log.
I20260927 14:51:32.206864 2016 tablet_bootstrap.cc:492] T 8b641035254348abafd39e71d62d7a70 P 124b623aa74049c7baeec32f6adcf3af: No bootstrap required, opened a new log
I20260927 14:51:32.206969 2016 ts_tablet_manager.cc:1397] T 8b641035254348abafd39e71d62d7a70 P 124b623aa74049c7baeec32f6adcf3af: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20260927 14:51:32.207435 2016 raft_consensus.cc:348] T 8b641035254348abafd39e71d62d7a70 P 124b623aa74049c7baeec32f6adcf3af [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c32f7b24fbc4b148527a0b1fedf8db8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43271 } } peers { permanent_uuid: "124b623aa74049c7baeec32f6adcf3af" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44477 } } peers { permanent_uuid: "060ee5e35c494ed4a668fcbf2660ff99" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45063 } }
I20260927 14:51:32.207585 2014 tablet_bootstrap.cc:492] T 8b641035254348abafd39e71d62d7a70 P 060ee5e35c494ed4a668fcbf2660ff99: Bootstrap starting.
I20260927 14:51:32.207557 2016 raft_consensus.cc:374] T 8b641035254348abafd39e71d62d7a70 P 124b623aa74049c7baeec32f6adcf3af [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 14:51:32.207864 2016 raft_consensus.cc:729] T 8b641035254348abafd39e71d62d7a70 P 124b623aa74049c7baeec32f6adcf3af [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 124b623aa74049c7baeec32f6adcf3af, State: Initialized, Role: FOLLOWER
I20260927 14:51:32.208389 2016 consensus_queue.cc:260] T 8b641035254348abafd39e71d62d7a70 P 124b623aa74049c7baeec32f6adcf3af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c32f7b24fbc4b148527a0b1fedf8db8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43271 } } peers { permanent_uuid: "124b623aa74049c7baeec32f6adcf3af" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44477 } } peers { permanent_uuid: "060ee5e35c494ed4a668fcbf2660ff99" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45063 } }
I20260927 14:51:32.208601 2014 tablet_bootstrap.cc:654] T 8b641035254348abafd39e71d62d7a70 P 060ee5e35c494ed4a668fcbf2660ff99: Neither blocks nor log segments found. Creating new log.
I20260927 14:51:32.208632 2013 tablet_bootstrap.cc:492] T 8b641035254348abafd39e71d62d7a70 P 9c32f7b24fbc4b148527a0b1fedf8db8: No bootstrap required, opened a new log
I20260927 14:51:32.208703 2013 ts_tablet_manager.cc:1397] T 8b641035254348abafd39e71d62d7a70 P 9c32f7b24fbc4b148527a0b1fedf8db8: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20260927 14:51:32.208724 2016 ts_tablet_manager.cc:1428] T 8b641035254348abafd39e71d62d7a70 P 124b623aa74049c7baeec32f6adcf3af: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20260927 14:51:32.209182 2013 raft_consensus.cc:348] T 8b641035254348abafd39e71d62d7a70 P 9c32f7b24fbc4b148527a0b1fedf8db8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c32f7b24fbc4b148527a0b1fedf8db8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43271 } } peers { permanent_uuid: "124b623aa74049c7baeec32f6adcf3af" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44477 } } peers { permanent_uuid: "060ee5e35c494ed4a668fcbf2660ff99" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45063 } }
I20260927 14:51:32.209338 2013 raft_consensus.cc:374] T 8b641035254348abafd39e71d62d7a70 P 9c32f7b24fbc4b148527a0b1fedf8db8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 14:51:32.209386 2013 raft_consensus.cc:729] T 8b641035254348abafd39e71d62d7a70 P 9c32f7b24fbc4b148527a0b1fedf8db8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9c32f7b24fbc4b148527a0b1fedf8db8, State: Initialized, Role: FOLLOWER
I20260927 14:51:32.209525 2013 consensus_queue.cc:260] T 8b641035254348abafd39e71d62d7a70 P 9c32f7b24fbc4b148527a0b1fedf8db8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c32f7b24fbc4b148527a0b1fedf8db8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43271 } } peers { permanent_uuid: "124b623aa74049c7baeec32f6adcf3af" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44477 } } peers { permanent_uuid: "060ee5e35c494ed4a668fcbf2660ff99" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45063 } }
I20260927 14:51:32.209805 2013 ts_tablet_manager.cc:1428] T 8b641035254348abafd39e71d62d7a70 P 9c32f7b24fbc4b148527a0b1fedf8db8: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260927 14:51:32.209754 2019 tablet_replica.cc:439] T 8b641035254348abafd39e71d62d7a70 P 9c32f7b24fbc4b148527a0b1fedf8db8: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c32f7b24fbc4b148527a0b1fedf8db8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43271 } } peers { permanent_uuid: "124b623aa74049c7baeec32f6adcf3af" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44477 } } peers { permanent_uuid: "060ee5e35c494ed4a668fcbf2660ff99" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45063 } } }
I20260927 14:51:32.209873 2019 tablet_replica.cc:442] T 8b641035254348abafd39e71d62d7a70 P 9c32f7b24fbc4b148527a0b1fedf8db8: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 14:51:32.210001 2032 tablet_replica.cc:439] T 8b641035254348abafd39e71d62d7a70 P 124b623aa74049c7baeec32f6adcf3af: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c32f7b24fbc4b148527a0b1fedf8db8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43271 } } peers { permanent_uuid: "124b623aa74049c7baeec32f6adcf3af" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44477 } } peers { permanent_uuid: "060ee5e35c494ed4a668fcbf2660ff99" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45063 } } }
I20260927 14:51:32.210150 2032 tablet_replica.cc:442] T 8b641035254348abafd39e71d62d7a70 P 124b623aa74049c7baeec32f6adcf3af: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 14:51:32.211356 2014 tablet_bootstrap.cc:492] T 8b641035254348abafd39e71d62d7a70 P 060ee5e35c494ed4a668fcbf2660ff99: No bootstrap required, opened a new log
I20260927 14:51:32.211457 2014 ts_tablet_manager.cc:1397] T 8b641035254348abafd39e71d62d7a70 P 060ee5e35c494ed4a668fcbf2660ff99: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260927 14:51:32.211890 2014 raft_consensus.cc:348] T 8b641035254348abafd39e71d62d7a70 P 060ee5e35c494ed4a668fcbf2660ff99 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c32f7b24fbc4b148527a0b1fedf8db8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43271 } } peers { permanent_uuid: "124b623aa74049c7baeec32f6adcf3af" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44477 } } peers { permanent_uuid: "060ee5e35c494ed4a668fcbf2660ff99" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45063 } }
I20260927 14:51:32.212004 2014 raft_consensus.cc:374] T 8b641035254348abafd39e71d62d7a70 P 060ee5e35c494ed4a668fcbf2660ff99 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 14:51:32.212045 2014 raft_consensus.cc:729] T 8b641035254348abafd39e71d62d7a70 P 060ee5e35c494ed4a668fcbf2660ff99 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 060ee5e35c494ed4a668fcbf2660ff99, State: Initialized, Role: FOLLOWER
I20260927 14:51:32.212177 2014 consensus_queue.cc:260] T 8b641035254348abafd39e71d62d7a70 P 060ee5e35c494ed4a668fcbf2660ff99 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c32f7b24fbc4b148527a0b1fedf8db8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43271 } } peers { permanent_uuid: "124b623aa74049c7baeec32f6adcf3af" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44477 } } peers { permanent_uuid: "060ee5e35c494ed4a668fcbf2660ff99" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45063 } }
I20260927 14:51:32.212436 2014 ts_tablet_manager.cc:1428] T 8b641035254348abafd39e71d62d7a70 P 060ee5e35c494ed4a668fcbf2660ff99: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260927 14:51:32.212406 2020 tablet_replica.cc:439] T 8b641035254348abafd39e71d62d7a70 P 060ee5e35c494ed4a668fcbf2660ff99: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c32f7b24fbc4b148527a0b1fedf8db8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43271 } } peers { permanent_uuid: "124b623aa74049c7baeec32f6adcf3af" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44477 } } peers { permanent_uuid: "060ee5e35c494ed4a668fcbf2660ff99" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45063 } } }
I20260927 14:51:32.212551 2020 tablet_replica.cc:442] T 8b641035254348abafd39e71d62d7a70 P 060ee5e35c494ed4a668fcbf2660ff99: This TxnStatusTablet replica's current role is: FOLLOWER
W20260927 14:51:32.293201 1711 tablet.cc:2369] T 8b641035254348abafd39e71d62d7a70 P 9c32f7b24fbc4b148527a0b1fedf8db8: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 14:51:32.520323 2019 consensus_queue.cc:1035] T af6c9e9c0c85420e8252fcdb9383e699 P 9c32f7b24fbc4b148527a0b1fedf8db8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "124b623aa74049c7baeec32f6adcf3af" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44477 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 14:51:32.521227 2020 raft_consensus.cc:482] T 8b641035254348abafd39e71d62d7a70 P 060ee5e35c494ed4a668fcbf2660ff99 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 14:51:32.521363 2020 raft_consensus.cc:504] T 8b641035254348abafd39e71d62d7a70 P 060ee5e35c494ed4a668fcbf2660ff99 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c32f7b24fbc4b148527a0b1fedf8db8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43271 } } peers { permanent_uuid: "124b623aa74049c7baeec32f6adcf3af" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44477 } } peers { permanent_uuid: "060ee5e35c494ed4a668fcbf2660ff99" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45063 } }
I20260927 14:51:32.522622 2020 leader_election.cc:288] T 8b641035254348abafd39e71d62d7a70 P 060ee5e35c494ed4a668fcbf2660ff99 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9c32f7b24fbc4b148527a0b1fedf8db8 (127.0.98.1:43271), 124b623aa74049c7baeec32f6adcf3af (127.0.98.2:44477)
I20260927 14:51:32.524266 2044 consensus_queue.cc:1035] T af6c9e9c0c85420e8252fcdb9383e699 P 9c32f7b24fbc4b148527a0b1fedf8db8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "060ee5e35c494ed4a668fcbf2660ff99" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45063 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 14:51:32.527675 2010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:51:32.523129 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:52783 (local address 127.0.98.2:44477)
0927 14:51:32.523248 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0927 14:51:32.523254 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 14:51:32.523497 (+ 243us) server_negotiation.cc:407] Connection header received
0927 14:51:32.523660 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:51:32.523663 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:51:32.523715 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:51:32.523784 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 14:51:32.524172 (+ 388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:51:32.524891 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:51:32.526455 (+ 1564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:51:32.527058 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:51:32.527134 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:51:32.527192 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0927 14:51:32.527503 (+ 311us) server_negotiation.cc:299] Negotiation successful
0927 14:51:32.527575 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260927 14:51:32.528072 2057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:51:32.522868 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:44477 (local address 127.0.98.3:52783)
0927 14:51:32.523411 (+ 543us) negotiation.cc:107] Waiting for socket to connect
0927 14:51:32.523427 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 14:51:32.523536 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:51:32.523893 (+ 357us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:51:32.523896 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:51:32.523926 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 14:51:32.524047 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:51:32.524054 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:51:32.525029 (+ 975us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:51:32.525032 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:51:32.526323 (+ 1291us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:51:32.526332 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:51:32.527281 (+ 949us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:51:32.527284 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:51:32.527368 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:51:32.527391 (+ 23us) client_negotiation.cc:769] Sending connection context
0927 14:51:32.527847 (+ 456us) client_negotiation.cc:240] Negotiation successful
0927 14:51:32.527936 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":481,"mutex_wait_us":50,"thread_start_us":56,"threads_started":1}
I20260927 14:51:32.528446 1804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8b641035254348abafd39e71d62d7a70" candidate_uuid: "060ee5e35c494ed4a668fcbf2660ff99" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "124b623aa74049c7baeec32f6adcf3af" is_pre_election: true
I20260927 14:51:32.528577 1804 raft_consensus.cc:2454] T 8b641035254348abafd39e71d62d7a70 P 124b623aa74049c7baeec32f6adcf3af [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 060ee5e35c494ed4a668fcbf2660ff99 in term 0.
I20260927 14:51:32.528926 1875 leader_election.cc:302] T 8b641035254348abafd39e71d62d7a70 P 060ee5e35c494ed4a668fcbf2660ff99 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 060ee5e35c494ed4a668fcbf2660ff99, 124b623aa74049c7baeec32f6adcf3af; no voters:
I20260927 14:51:32.529204 2058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:51:32.523143 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:36313 (local address 127.0.98.1:43271)
0927 14:51:32.524163 (+ 1020us) server_negotiation.cc:206] Beginning negotiation
0927 14:51:32.524167 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 14:51:32.524178 (+ 11us) server_negotiation.cc:407] Connection header received
0927 14:51:32.524217 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:51:32.524219 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:51:32.524261 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:51:32.524334 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 14:51:32.524761 (+ 427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:51:32.525519 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:51:32.528043 (+ 2524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:51:32.528668 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:51:32.528777 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:51:32.528831 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0927 14:51:32.529001 (+ 170us) server_negotiation.cc:299] Negotiation successful
0927 14:51:32.529094 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":936,"thread_start_us":89,"threads_started":1}
I20260927 14:51:32.529204 2056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:51:32.522801 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:43271 (local address 127.0.98.3:36313)
0927 14:51:32.523264 (+ 463us) negotiation.cc:107] Waiting for socket to connect
0927 14:51:32.523280 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 14:51:32.523383 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:51:32.524518 (+ 1135us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:51:32.524521 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:51:32.524538 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 14:51:32.524638 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:51:32.524643 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:51:32.525642 (+ 999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:51:32.525645 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:51:32.527018 (+ 1373us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:51:32.527909 (+ 891us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:51:32.528773 (+ 864us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:51:32.528777 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:51:32.528877 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:51:32.528901 (+ 24us) client_negotiation.cc:769] Sending connection context
0927 14:51:32.528988 (+ 87us) client_negotiation.cc:240] Negotiation successful
0927 14:51:32.529053 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":380,"mutex_wait_us":21,"thread_start_us":76,"threads_started":1}
I20260927 14:51:32.529518 2060 raft_consensus.cc:2781] T 8b641035254348abafd39e71d62d7a70 P 060ee5e35c494ed4a668fcbf2660ff99 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 14:51:32.529585 2060 raft_consensus.cc:482] T 8b641035254348abafd39e71d62d7a70 P 060ee5e35c494ed4a668fcbf2660ff99 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 14:51:32.529624 2060 raft_consensus.cc:3037] T 8b641035254348abafd39e71d62d7a70 P 060ee5e35c494ed4a668fcbf2660ff99 [term 0 FOLLOWER]: Advancing to term 1
I20260927 14:51:32.529668 1654 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8b641035254348abafd39e71d62d7a70" candidate_uuid: "060ee5e35c494ed4a668fcbf2660ff99" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c32f7b24fbc4b148527a0b1fedf8db8" is_pre_election: true
I20260927 14:51:32.529789 1654 raft_consensus.cc:2454] T 8b641035254348abafd39e71d62d7a70 P 9c32f7b24fbc4b148527a0b1fedf8db8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 060ee5e35c494ed4a668fcbf2660ff99 in term 0.
I20260927 14:51:32.530706 2060 raft_consensus.cc:504] T 8b641035254348abafd39e71d62d7a70 P 060ee5e35c494ed4a668fcbf2660ff99 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c32f7b24fbc4b148527a0b1fedf8db8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43271 } } peers { permanent_uuid: "124b623aa74049c7baeec32f6adcf3af" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44477 } } peers { permanent_uuid: "060ee5e35c494ed4a668fcbf2660ff99" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45063 } }
I20260927 14:51:32.531083 2060 leader_election.cc:288] T 8b641035254348abafd39e71d62d7a70 P 060ee5e35c494ed4a668fcbf2660ff99 [CANDIDATE]: Term 1 election: Requested vote from peers 9c32f7b24fbc4b148527a0b1fedf8db8 (127.0.98.1:43271), 124b623aa74049c7baeec32f6adcf3af (127.0.98.2:44477)
I20260927 14:51:32.531351 1654 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8b641035254348abafd39e71d62d7a70" candidate_uuid: "060ee5e35c494ed4a668fcbf2660ff99" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c32f7b24fbc4b148527a0b1fedf8db8"
I20260927 14:51:32.531445 1654 raft_consensus.cc:3037] T 8b641035254348abafd39e71d62d7a70 P 9c32f7b24fbc4b148527a0b1fedf8db8 [term 0 FOLLOWER]: Advancing to term 1
I20260927 14:51:32.531551 1804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8b641035254348abafd39e71d62d7a70" candidate_uuid: "060ee5e35c494ed4a668fcbf2660ff99" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "124b623aa74049c7baeec32f6adcf3af"
I20260927 14:51:32.531652 1804 raft_consensus.cc:3037] T 8b641035254348abafd39e71d62d7a70 P 124b623aa74049c7baeec32f6adcf3af [term 0 FOLLOWER]: Advancing to term 1
I20260927 14:51:32.532275 1654 raft_consensus.cc:2454] T 8b641035254348abafd39e71d62d7a70 P 9c32f7b24fbc4b148527a0b1fedf8db8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 060ee5e35c494ed4a668fcbf2660ff99 in term 1.
I20260927 14:51:32.532823 1804 raft_consensus.cc:2454] T 8b641035254348abafd39e71d62d7a70 P 124b623aa74049c7baeec32f6adcf3af [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 060ee5e35c494ed4a668fcbf2660ff99 in term 1.
I20260927 14:51:32.533200 1875 leader_election.cc:302] T 8b641035254348abafd39e71d62d7a70 P 060ee5e35c494ed4a668fcbf2660ff99 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 060ee5e35c494ed4a668fcbf2660ff99, 124b623aa74049c7baeec32f6adcf3af; no voters:
I20260927 14:51:32.533371 2060 raft_consensus.cc:2781] T 8b641035254348abafd39e71d62d7a70 P 060ee5e35c494ed4a668fcbf2660ff99 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 14:51:32.533466 2060 raft_consensus.cc:686] T 8b641035254348abafd39e71d62d7a70 P 060ee5e35c494ed4a668fcbf2660ff99 [term 1 LEADER]: Becoming Leader. State: Replica: 060ee5e35c494ed4a668fcbf2660ff99, State: Running, Role: LEADER
I20260927 14:51:32.533684 2060 consensus_queue.cc:237] T 8b641035254348abafd39e71d62d7a70 P 060ee5e35c494ed4a668fcbf2660ff99 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c32f7b24fbc4b148527a0b1fedf8db8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43271 } } peers { permanent_uuid: "124b623aa74049c7baeec32f6adcf3af" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44477 } } peers { permanent_uuid: "060ee5e35c494ed4a668fcbf2660ff99" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45063 } }
I20260927 14:51:32.534829 2020 tablet_replica.cc:439] T 8b641035254348abafd39e71d62d7a70 P 060ee5e35c494ed4a668fcbf2660ff99: TxnStatusTablet state changed. Reason: New leader 060ee5e35c494ed4a668fcbf2660ff99. Latest consensus state: current_term: 1 leader_uuid: "060ee5e35c494ed4a668fcbf2660ff99" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c32f7b24fbc4b148527a0b1fedf8db8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43271 } } peers { permanent_uuid: "124b623aa74049c7baeec32f6adcf3af" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44477 } } peers { permanent_uuid: "060ee5e35c494ed4a668fcbf2660ff99" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45063 } } }
I20260927 14:51:32.534950 2020 tablet_replica.cc:442] T 8b641035254348abafd39e71d62d7a70 P 060ee5e35c494ed4a668fcbf2660ff99: This TxnStatusTablet replica's current role is: LEADER
I20260927 14:51:32.536033 1506 catalog_manager.cc:5495] T 8b641035254348abafd39e71d62d7a70 P 060ee5e35c494ed4a668fcbf2660ff99 reported cstate change: term changed from 0 to 1, leader changed from <none> to 060ee5e35c494ed4a668fcbf2660ff99 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "060ee5e35c494ed4a668fcbf2660ff99" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c32f7b24fbc4b148527a0b1fedf8db8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43271 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "124b623aa74049c7baeec32f6adcf3af" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44477 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "060ee5e35c494ed4a668fcbf2660ff99" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45063 } health_report { overall_health: HEALTHY } } }
I20260927 14:51:32.537204 2062 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260927 14:51:32.537352 2062 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260927 14:51:32.575256 2009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:51:32.571619 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57516 (local address 127.0.98.3:45063)
0927 14:51:32.571749 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0927 14:51:32.571754 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 14:51:32.571828 (+ 74us) server_negotiation.cc:407] Connection header received
0927 14:51:32.572074 (+ 246us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:51:32.572077 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:51:32.572130 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:51:32.572229 (+ 99us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 14:51:32.572465 (+ 236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:51:32.572980 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:51:32.573833 (+ 853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:51:32.574053 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:51:32.574157 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:51:32.574454 (+ 297us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 14:51:32.574873 (+ 419us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 14:51:32.574994 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0927 14:51:32.575065 (+ 71us) server_negotiation.cc:299] Negotiation successful
0927 14:51:32.575114 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260927 14:51:32.575256 2051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:51:32.571504 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:45063 (local address 127.0.0.1:57516)
0927 14:51:32.571765 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0927 14:51:32.571782 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 14:51:32.571905 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:51:32.572238 (+ 333us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:51:32.572241 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:51:32.572250 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 14:51:32.572355 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:51:32.572361 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:51:32.573100 (+ 739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:51:32.573106 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:51:32.573719 (+ 613us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:51:32.573727 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:51:32.574158 (+ 431us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:51:32.574161 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:51:32.574252 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:51:32.574303 (+ 51us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 14:51:32.574969 (+ 666us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 14:51:32.574985 (+ 16us) client_negotiation.cc:769] Sending connection context
0927 14:51:32.575055 (+ 70us) client_negotiation.cc:240] Negotiation successful
0927 14:51:32.575141 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":139}
I20260927 14:51:32.578295 1654 raft_consensus.cc:1261] T 8b641035254348abafd39e71d62d7a70 P 9c32f7b24fbc4b148527a0b1fedf8db8 [term 1 FOLLOWER]: Refusing update from remote peer 060ee5e35c494ed4a668fcbf2660ff99: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 14:51:32.578347 1804 raft_consensus.cc:1261] T 8b641035254348abafd39e71d62d7a70 P 124b623aa74049c7baeec32f6adcf3af [term 1 FOLLOWER]: Refusing update from remote peer 060ee5e35c494ed4a668fcbf2660ff99: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 14:51:32.578783 2020 consensus_queue.cc:1035] T 8b641035254348abafd39e71d62d7a70 P 060ee5e35c494ed4a668fcbf2660ff99 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c32f7b24fbc4b148527a0b1fedf8db8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43271 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 14:51:32.578950 2060 consensus_queue.cc:1035] T 8b641035254348abafd39e71d62d7a70 P 060ee5e35c494ed4a668fcbf2660ff99 [LEADER]: Connected to new peer: Peer: permanent_uuid: "124b623aa74049c7baeec32f6adcf3af" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44477 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 14:51:32.580727 2032 tablet_replica.cc:439] T 8b641035254348abafd39e71d62d7a70 P 124b623aa74049c7baeec32f6adcf3af: TxnStatusTablet state changed. Reason: New leader 060ee5e35c494ed4a668fcbf2660ff99. Latest consensus state: current_term: 1 leader_uuid: "060ee5e35c494ed4a668fcbf2660ff99" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c32f7b24fbc4b148527a0b1fedf8db8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43271 } } peers { permanent_uuid: "124b623aa74049c7baeec32f6adcf3af" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44477 } } peers { permanent_uuid: "060ee5e35c494ed4a668fcbf2660ff99" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45063 } } }
I20260927 14:51:32.580860 2032 tablet_replica.cc:442] T 8b641035254348abafd39e71d62d7a70 P 124b623aa74049c7baeec32f6adcf3af: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 14:51:32.581321 2044 tablet_replica.cc:439] T 8b641035254348abafd39e71d62d7a70 P 9c32f7b24fbc4b148527a0b1fedf8db8: TxnStatusTablet state changed. Reason: New leader 060ee5e35c494ed4a668fcbf2660ff99. Latest consensus state: current_term: 1 leader_uuid: "060ee5e35c494ed4a668fcbf2660ff99" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c32f7b24fbc4b148527a0b1fedf8db8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43271 } } peers { permanent_uuid: "124b623aa74049c7baeec32f6adcf3af" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44477 } } peers { permanent_uuid: "060ee5e35c494ed4a668fcbf2660ff99" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45063 } } }
I20260927 14:51:32.581453 2044 tablet_replica.cc:442] T 8b641035254348abafd39e71d62d7a70 P 9c32f7b24fbc4b148527a0b1fedf8db8: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 14:51:32.583673 2044 tablet_replica.cc:439] T 8b641035254348abafd39e71d62d7a70 P 9c32f7b24fbc4b148527a0b1fedf8db8: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "060ee5e35c494ed4a668fcbf2660ff99" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c32f7b24fbc4b148527a0b1fedf8db8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43271 } } peers { permanent_uuid: "124b623aa74049c7baeec32f6adcf3af" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44477 } } peers { permanent_uuid: "060ee5e35c494ed4a668fcbf2660ff99" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45063 } } }
I20260927 14:51:32.583793 2044 tablet_replica.cc:442] T 8b641035254348abafd39e71d62d7a70 P 9c32f7b24fbc4b148527a0b1fedf8db8: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 14:51:32.583751 2027 mvcc.cc:205] Tried to move back new op lower bound from 7333972756795768832 to 7333972756620304384. Current Snapshot: MvccSnapshot[applied={T|T < 7333972756795768832}]
I20260927 14:51:32.584369 2032 tablet_replica.cc:439] T 8b641035254348abafd39e71d62d7a70 P 124b623aa74049c7baeec32f6adcf3af: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "060ee5e35c494ed4a668fcbf2660ff99" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c32f7b24fbc4b148527a0b1fedf8db8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43271 } } peers { permanent_uuid: "124b623aa74049c7baeec32f6adcf3af" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44477 } } peers { permanent_uuid: "060ee5e35c494ed4a668fcbf2660ff99" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45063 } } }
I20260927 14:51:32.584507 2032 tablet_replica.cc:442] T 8b641035254348abafd39e71d62d7a70 P 124b623aa74049c7baeec32f6adcf3af: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 14:51:32.584682 2068 tablet_replica.cc:439] T 8b641035254348abafd39e71d62d7a70 P 060ee5e35c494ed4a668fcbf2660ff99: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "060ee5e35c494ed4a668fcbf2660ff99" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c32f7b24fbc4b148527a0b1fedf8db8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43271 } } peers { permanent_uuid: "124b623aa74049c7baeec32f6adcf3af" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44477 } } peers { permanent_uuid: "060ee5e35c494ed4a668fcbf2660ff99" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45063 } } }
I20260927 14:51:32.584794 2068 tablet_replica.cc:442] T 8b641035254348abafd39e71d62d7a70 P 060ee5e35c494ed4a668fcbf2660ff99: This TxnStatusTablet replica's current role is: LEADER
I20260927 14:51:32.586447 2060 tablet_replica.cc:439] T 8b641035254348abafd39e71d62d7a70 P 060ee5e35c494ed4a668fcbf2660ff99: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "060ee5e35c494ed4a668fcbf2660ff99" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c32f7b24fbc4b148527a0b1fedf8db8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43271 } } peers { permanent_uuid: "124b623aa74049c7baeec32f6adcf3af" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44477 } } peers { permanent_uuid: "060ee5e35c494ed4a668fcbf2660ff99" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45063 } } }
I20260927 14:51:32.586555 2060 tablet_replica.cc:442] T 8b641035254348abafd39e71d62d7a70 P 060ee5e35c494ed4a668fcbf2660ff99: This TxnStatusTablet replica's current role is: LEADER
I20260927 14:51:32.596894 2058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:51:32.590401 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53298 (local address 127.0.98.1:43271)
0927 14:51:32.590498 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0927 14:51:32.590502 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 14:51:32.593385 (+ 2883us) server_negotiation.cc:407] Connection header received
0927 14:51:32.593590 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:51:32.593593 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:51:32.593645 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:51:32.593746 (+ 101us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 14:51:32.594037 (+ 291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:51:32.594563 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:51:32.595515 (+ 952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:51:32.595732 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:51:32.595834 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:51:32.596100 (+ 266us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 14:51:32.596521 (+ 421us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 14:51:32.596644 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0927 14:51:32.596707 (+ 63us) server_negotiation.cc:299] Negotiation successful
0927 14:51:32.596759 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260927 14:51:32.607475 2077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:51:32.603512 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:45063 (local address 127.0.98.1:56849)
0927 14:51:32.603800 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0927 14:51:32.603813 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 14:51:32.603900 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:51:32.604144 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:51:32.604146 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:51:32.604153 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 14:51:32.604233 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:51:32.604237 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:51:32.605195 (+ 958us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:51:32.605198 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:51:32.606168 (+ 970us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:51:32.606177 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:51:32.606725 (+ 548us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:51:32.606728 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:51:32.606792 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:51:32.606832 (+ 40us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 14:51:32.607156 (+ 324us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 14:51:32.607165 (+ 9us) client_negotiation.cc:769] Sending connection context
0927 14:51:32.607243 (+ 78us) client_negotiation.cc:240] Negotiation successful
0927 14:51:32.607321 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":105,"threads_started":1}
I20260927 14:51:32.607462 2009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:51:32.603615 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:56849 (local address 127.0.98.3:45063)
0927 14:51:32.603749 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0927 14:51:32.603753 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 14:51:32.603887 (+ 134us) server_negotiation.cc:407] Connection header received
0927 14:51:32.603985 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:51:32.603988 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:51:32.604039 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:51:32.604121 (+ 82us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 14:51:32.604325 (+ 204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:51:32.605103 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:51:32.606276 (+ 1173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:51:32.606616 (+ 340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:51:32.606711 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:51:32.606989 (+ 278us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 14:51:32.607071 (+ 82us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 14:51:32.607152 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0927 14:51:32.607268 (+ 116us) server_negotiation.cc:299] Negotiation successful
0927 14:51:32.607318 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260927 14:51:32.634438 2058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:51:32.630737 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:54661 (local address 127.0.98.1:43271)
0927 14:51:32.630850 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0927 14:51:32.630854 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 14:51:32.631041 (+ 187us) server_negotiation.cc:407] Connection header received
0927 14:51:32.631202 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:51:32.631205 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:51:32.631250 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:51:32.631353 (+ 103us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 14:51:32.631648 (+ 295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:51:32.632333 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:51:32.633365 (+ 1032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:51:32.633593 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:51:32.633706 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:51:32.633986 (+ 280us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 14:51:32.634066 (+ 80us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 14:51:32.634143 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0927 14:51:32.634230 (+ 87us) server_negotiation.cc:299] Negotiation successful
0927 14:51:32.634276 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260927 14:51:32.634876 2080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:51:32.630676 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:43271 (local address 127.0.98.3:54661)
0927 14:51:32.630966 (+ 290us) negotiation.cc:107] Waiting for socket to connect
0927 14:51:32.630978 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 14:51:32.631101 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:51:32.631369 (+ 268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:51:32.631372 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:51:32.631379 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 14:51:32.631512 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:51:32.631518 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:51:32.632467 (+ 949us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:51:32.632470 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:51:32.633249 (+ 779us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:51:32.633258 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:51:32.633714 (+ 456us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:51:32.633717 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:51:32.633814 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:51:32.633854 (+ 40us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 14:51:32.634147 (+ 293us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 14:51:32.634155 (+ 8us) client_negotiation.cc:769] Sending connection context
0927 14:51:32.634221 (+ 66us) client_negotiation.cc:240] Negotiation successful
0927 14:51:32.634296 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":173,"thread_start_us":99,"threads_started":1}
I20260927 14:51:32.664471 1506 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:53370:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 14:51:32.664654 1506 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:53370:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 14:51:32.667241 1506 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a03aa1549f69443084f0c6a8c1b06bd6: Sending DeleteTablet for 3 replicas of tablet af6c9e9c0c85420e8252fcdb9383e699
I20260927 14:51:32.667852 1644 tablet_service.cc:1514] Processing DeleteTablet for tablet af6c9e9c0c85420e8252fcdb9383e699 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 14:51:32 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:53292
I20260927 14:51:32.668321 1782 tablet_service.cc:1514] Processing DeleteTablet for tablet af6c9e9c0c85420e8252fcdb9383e699 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 14:51:32 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:52340
I20260927 14:51:32.668633 2084 tablet_replica.cc:330] T af6c9e9c0c85420e8252fcdb9383e699 P 124b623aa74049c7baeec32f6adcf3af: stopping tablet replica
I20260927 14:51:32.668751 2084 raft_consensus.cc:2229] T af6c9e9c0c85420e8252fcdb9383e699 P 124b623aa74049c7baeec32f6adcf3af [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 14:51:32.668861 2084 raft_consensus.cc:2258] T af6c9e9c0c85420e8252fcdb9383e699 P 124b623aa74049c7baeec32f6adcf3af [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 14:51:32.669592 1924 tablet_service.cc:1514] Processing DeleteTablet for tablet af6c9e9c0c85420e8252fcdb9383e699 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 14:51:32 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:57502
I20260927 14:51:32.669618 2084 ts_tablet_manager.cc:1905] T af6c9e9c0c85420e8252fcdb9383e699 P 124b623aa74049c7baeec32f6adcf3af: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 14:51:32.671178 2083 tablet_replica.cc:330] T af6c9e9c0c85420e8252fcdb9383e699 P 9c32f7b24fbc4b148527a0b1fedf8db8: stopping tablet replica
I20260927 14:51:32.671358 2083 raft_consensus.cc:2229] T af6c9e9c0c85420e8252fcdb9383e699 P 9c32f7b24fbc4b148527a0b1fedf8db8 [term 1 LEADER]: Raft consensus shutting down.
I20260927 14:51:32.671720 2083 raft_consensus.cc:2258] T af6c9e9c0c85420e8252fcdb9383e699 P 9c32f7b24fbc4b148527a0b1fedf8db8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 14:51:32.672452 2083 ts_tablet_manager.cc:1905] T af6c9e9c0c85420e8252fcdb9383e699 P 9c32f7b24fbc4b148527a0b1fedf8db8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 14:51:32.672586 2084 ts_tablet_manager.cc:1918] T af6c9e9c0c85420e8252fcdb9383e699 P 124b623aa74049c7baeec32f6adcf3af: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 14:51:32.672672 2084 log.cc:1196] T af6c9e9c0c85420e8252fcdb9383e699 P 124b623aa74049c7baeec32f6adcf3af: Deleting WAL directory at /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790520684602251-392-0/minicluster-data/ts-1/wal/wals/af6c9e9c0c85420e8252fcdb9383e699
I20260927 14:51:32.672931 2084 ts_tablet_manager.cc:1939] T af6c9e9c0c85420e8252fcdb9383e699 P 124b623aa74049c7baeec32f6adcf3af: Deleting consensus metadata
I20260927 14:51:32.673637 1491 catalog_manager.cc:4841] TS 124b623aa74049c7baeec32f6adcf3af (127.0.98.2:44477): tablet af6c9e9c0c85420e8252fcdb9383e699 (table test-table [id=a38a3e6c14c742718d93fed486a865ee]) successfully deleted
I20260927 14:51:32.675594 2083 ts_tablet_manager.cc:1918] T af6c9e9c0c85420e8252fcdb9383e699 P 9c32f7b24fbc4b148527a0b1fedf8db8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 14:51:32.675792 2086 tablet_replica.cc:330] T af6c9e9c0c85420e8252fcdb9383e699 P 060ee5e35c494ed4a668fcbf2660ff99: stopping tablet replica
I20260927 14:51:32.675921 2086 raft_consensus.cc:2229] T af6c9e9c0c85420e8252fcdb9383e699 P 060ee5e35c494ed4a668fcbf2660ff99 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 14:51:32.676030 2086 raft_consensus.cc:2258] T af6c9e9c0c85420e8252fcdb9383e699 P 060ee5e35c494ed4a668fcbf2660ff99 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 14:51:32.676292 2083 log.cc:1196] T af6c9e9c0c85420e8252fcdb9383e699 P 9c32f7b24fbc4b148527a0b1fedf8db8: Deleting WAL directory at /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790520684602251-392-0/minicluster-data/ts-0/wal/wals/af6c9e9c0c85420e8252fcdb9383e699
I20260927 14:51:32.676643 2083 ts_tablet_manager.cc:1939] T af6c9e9c0c85420e8252fcdb9383e699 P 9c32f7b24fbc4b148527a0b1fedf8db8: Deleting consensus metadata
I20260927 14:51:32.676666 2086 ts_tablet_manager.cc:1905] T af6c9e9c0c85420e8252fcdb9383e699 P 060ee5e35c494ed4a668fcbf2660ff99: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 14:51:32.677424 1492 catalog_manager.cc:4841] TS 9c32f7b24fbc4b148527a0b1fedf8db8 (127.0.98.1:43271): tablet af6c9e9c0c85420e8252fcdb9383e699 (table test-table [id=a38a3e6c14c742718d93fed486a865ee]) successfully deleted
I20260927 14:51:32.679293 2086 ts_tablet_manager.cc:1918] T af6c9e9c0c85420e8252fcdb9383e699 P 060ee5e35c494ed4a668fcbf2660ff99: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 14:51:32.679371 2086 log.cc:1196] T af6c9e9c0c85420e8252fcdb9383e699 P 060ee5e35c494ed4a668fcbf2660ff99: Deleting WAL directory at /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790520684602251-392-0/minicluster-data/ts-2/wal/wals/af6c9e9c0c85420e8252fcdb9383e699
I20260927 14:51:32.679623 2086 ts_tablet_manager.cc:1939] T af6c9e9c0c85420e8252fcdb9383e699 P 060ee5e35c494ed4a668fcbf2660ff99: Deleting consensus metadata
I20260927 14:51:32.680226 1494 catalog_manager.cc:4841] TS 060ee5e35c494ed4a668fcbf2660ff99 (127.0.98.3:45063): tablet af6c9e9c0c85420e8252fcdb9383e699 (table test-table [id=a38a3e6c14c742718d93fed486a865ee]) successfully deleted
Sep 27 14:51:32 dist-test-slave-xk1g krb5kdc[1462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790520691, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260927 14:51:32.690008 2058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:51:32.670612 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53314 (local address 127.0.98.1:43271)
0927 14:51:32.670735 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0927 14:51:32.670739 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 14:51:32.677944 (+ 7205us) server_negotiation.cc:407] Connection header received
0927 14:51:32.678171 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:51:32.678174 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:51:32.678227 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:51:32.678282 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:51:32.679281 (+ 999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:51:32.680038 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:51:32.681351 (+ 1313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:51:32.681698 (+ 347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:51:32.681802 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:51:32.685713 (+ 3911us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:51:32.685724 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:51:32.685740 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 14:51:32.685767 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:51:32.687881 (+ 2114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:51:32.688386 (+ 505us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:51:32.688392 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:51:32.688394 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:51:32.688489 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:51:32.688773 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:51:32.688776 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:51:32.688778 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:51:32.689140 (+ 362us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:51:32.689334 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0927 14:51:32.689632 (+ 298us) server_negotiation.cc:299] Negotiation successful
0927 14:51:32.689822 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
W20260927 14:51:32.690439 1704 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:53314
I20260927 14:51:32.702287 2089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:51:32.692248 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53382 (local address 127.0.98.62:40621)
0927 14:51:32.692738 (+ 490us) server_negotiation.cc:206] Beginning negotiation
0927 14:51:32.692743 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 14:51:32.692950 (+ 207us) server_negotiation.cc:407] Connection header received
0927 14:51:32.693117 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:51:32.693121 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:51:32.693176 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:51:32.693269 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:51:32.694489 (+ 1220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:51:32.695444 (+ 955us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:51:32.696635 (+ 1191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:51:32.696988 (+ 353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:51:32.697134 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:51:32.698765 (+ 1631us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:51:32.698769 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:51:32.698772 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 14:51:32.698801 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:51:32.700565 (+ 1764us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:51:32.701014 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:51:32.701018 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:51:32.701020 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:51:32.701156 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:51:32.701429 (+ 273us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:51:32.701432 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:51:32.701434 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:51:32.701644 (+ 210us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:51:32.701775 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0927 14:51:32.701987 (+ 212us) server_negotiation.cc:299] Negotiation successful
0927 14:51:32.702110 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":414,"thread_start_us":76,"threads_started":1}
W20260927 14:51:32.702653 1506 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:53382
I20260927 14:51:32.704001 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6936m5/build/debug/bin/kudu with pid 1579
I20260927 14:51:32.712028 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6936m5/build/debug/bin/kudu with pid 1719
I20260927 14:51:32.719159 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6936m5/build/debug/bin/kudu with pid 1859
I20260927 14:51:32.726161 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6936m5/build/debug/bin/kudu with pid 1475
2026-09-27T14:51:32Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (3140 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790520684602251-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 14:51:32 dist-test-slave-xk1g krb5kdc[2093](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 14:51:32 dist-test-slave-xk1g krb5kdc[2093](info): set up 2 sockets
Sep 27 14:51:32 dist-test-slave-xk1g krb5kdc[2093](info): commencing operation
krb5kdc: starting...
W20260927 14:51:33.777130 392 mini_kdc.cc:121] Time spent starting KDC: real 1.029s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790520684602251-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790520684602251-392-0/krb5kdc/test-user.keytab.
Sep 27 14:51:33 dist-test-slave-xk1g krb5kdc[2093](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790520693, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T14:51:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T14:51:33Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790520684602251-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790520684602251-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 14:51:33.912611 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6936m5/build/debug/bin/kudu
/tmp/dist-test-task6936m5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790520684602251-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790520684602251-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790520684602251-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790520684602251-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40817
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790520684602251-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41793
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:40817
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790520684602251-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790520684602251-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790520684602251-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 14:51:34.021641 2106 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 14:51:34.021925 2106 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 14:51:34.021971 2106 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 14:51:34.026096 2106 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 14:51:34.026170 2106 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260927 14:51:34.026187 2106 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 14:51:34.026206 2106 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 14:51:34.026222 2106 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 14:51:34.026278 2106 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 14:51:34.031881 2106 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41793
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790520684602251-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790520684602251-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:40817
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790520684602251-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40817
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790520684602251-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2106
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790520684602251-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:15:17 UTC on ccycloud.quasar-ledxfe.root.comops.site
build id 4676134
I20260927 14:51:34.033021 2106 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 14:51:34.034098 2106 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 14:51:34.042390 2114 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 14:51:34.042392 2112 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 14:51:34.042404 2111 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 14:51:34.042591 2106 server_base.cc:1034] running on GCE node
I20260927 14:51:34.043258 2106 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 14:51:34.044350 2106 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 14:51:34.045553 2106 hybrid_clock.cc:648] HybridClock initialized: now 1790520694045534 us; error 32 us; skew 500 ppm
Sep 27 14:51:34 dist-test-slave-xk1g krb5kdc[2093](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790520694, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 14:51:34.048415 2106 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 14:51:34.049278 2106 webserver.cc:460] Webserver started at http://127.0.98.62:34403/ using document root <none> and password file <none>
I20260927 14:51:34.049839 2106 fs_manager.cc:359] Metadata directory not provided
I20260927 14:51:34.049933 2106 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 14:51:34.050163 2106 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 14:51:34.052173 2106 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790520684602251-392-0/minicluster-data/master-0/data/instance:
uuid: "4939c7ba397e4ab48e64602dd2f1047c"
format_stamp: "Formatted at 2026-09-27 14:51:34 on dist-test-slave-xk1g"
server_key: "f0304cca2e3c95644af2ab436b4ebd97"
server_key_iv: "a7bd7213383016c90ef79f9ebd1b2bd7"
server_key_version: "encryptionkey@0"
I20260927 14:51:34.052729 2106 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790520684602251-392-0/minicluster-data/master-0/wal/instance:
uuid: "4939c7ba397e4ab48e64602dd2f1047c"
format_stamp: "Formatted at 2026-09-27 14:51:34 on dist-test-slave-xk1g"
server_key: "f0304cca2e3c95644af2ab436b4ebd97"
server_key_iv: "a7bd7213383016c90ef79f9ebd1b2bd7"
server_key_version: "encryptionkey@0"
I20260927 14:51:34.056270 2106 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.003s
I20260927 14:51:34.058758 2121 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 14:51:34.060034 2106 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 14:51:34.060181 2106 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790520684602251-392-0/minicluster-data/master-0/data,/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790520684602251-392-0/minicluster-data/master-0/wal
uuid: "4939c7ba397e4ab48e64602dd2f1047c"
format_stamp: "Formatted at 2026-09-27 14:51:34 on dist-test-slave-xk1g"
server_key: "f0304cca2e3c95644af2ab436b4ebd97"
server_key_iv: "a7bd7213383016c90ef79f9ebd1b2bd7"
server_key_version: "encryptionkey@0"
I20260927 14:51:34.060297 2106 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790520684602251-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790520684602251-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790520684602251-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 14:51:34.070976 2106 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 14:51:34.074208 2106 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 14:51:34.074402 2106 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 14:51:34.081773 2106 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:40817
I20260927 14:51:34.081794 2183 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:40817 every 8 connection(s)
I20260927 14:51:34.082680 2106 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790520684602251-392-0/minicluster-data/master-0/data/info.pb
I20260927 14:51:34.085439 2184 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 14:51:34.088529 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6936m5/build/debug/bin/kudu as pid 2106
I20260927 14:51:34.088624 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790520684602251-392-0/minicluster-data/master-0/wal/instance
I20260927 14:51:34.088886 392 external_mini_cluster.cc:1428] Setting key da1a66e00416bf4e60d88169416497bd
I20260927 14:51:34.091437 2184 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4939c7ba397e4ab48e64602dd2f1047c: Bootstrap starting.
I20260927 14:51:34.093658 2184 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4939c7ba397e4ab48e64602dd2f1047c: Neither blocks nor log segments found. Creating new log.
I20260927 14:51:34.094400 2184 log.cc:824] T 00000000000000000000000000000000 P 4939c7ba397e4ab48e64602dd2f1047c: Log is configured to *not* fsync() on all Append() calls
I20260927 14:51:34.096340 2184 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4939c7ba397e4ab48e64602dd2f1047c: No bootstrap required, opened a new log
Sep 27 14:51:34 dist-test-slave-xk1g krb5kdc[2093](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790520693, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 14:51:34.100611 2184 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4939c7ba397e4ab48e64602dd2f1047c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4939c7ba397e4ab48e64602dd2f1047c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40817 } }
I20260927 14:51:34.100912 2184 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4939c7ba397e4ab48e64602dd2f1047c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 14:51:34.100991 2184 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4939c7ba397e4ab48e64602dd2f1047c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4939c7ba397e4ab48e64602dd2f1047c, State: Initialized, Role: FOLLOWER
I20260927 14:51:34.101468 2184 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4939c7ba397e4ab48e64602dd2f1047c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4939c7ba397e4ab48e64602dd2f1047c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40817 } }
I20260927 14:51:34.101631 2184 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4939c7ba397e4ab48e64602dd2f1047c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 14:51:34.101716 2184 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4939c7ba397e4ab48e64602dd2f1047c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 14:51:34.101823 2184 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4939c7ba397e4ab48e64602dd2f1047c [term 0 FOLLOWER]: Advancing to term 1
I20260927 14:51:34.102742 2184 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4939c7ba397e4ab48e64602dd2f1047c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4939c7ba397e4ab48e64602dd2f1047c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40817 } }
I20260927 14:51:34.102977 2187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:51:34.090241 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50578 (local address 127.0.98.62:40817)
0927 14:51:34.090677 (+ 436us) server_negotiation.cc:206] Beginning negotiation
0927 14:51:34.090691 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 14:51:34.090726 (+ 35us) server_negotiation.cc:407] Connection header received
0927 14:51:34.091339 (+ 613us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:51:34.091355 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:51:34.091717 (+ 362us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:51:34.092047 (+ 330us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:51:34.092807 (+ 760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:51:34.093887 (+ 1080us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:51:34.094757 (+ 870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:51:34.095226 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:51:34.095387 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:51:34.098186 (+ 2799us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:51:34.098191 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:51:34.098205 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 14:51:34.098232 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:51:34.099893 (+ 1661us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:51:34.100379 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:51:34.100384 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:51:34.100389 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:51:34.100494 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:51:34.100772 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:51:34.100774 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:51:34.100775 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:51:34.101334 (+ 559us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:51:34.101494 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0927 14:51:34.101841 (+ 347us) server_negotiation.cc:299] Negotiation successful
0927 14:51:34.102177 (+ 336us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":297,"thread_start_us":138,"threads_started":1}
I20260927 14:51:34.103112 2184 leader_election.cc:302] T 00000000000000000000000000000000 P 4939c7ba397e4ab48e64602dd2f1047c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4939c7ba397e4ab48e64602dd2f1047c; no voters:
I20260927 14:51:34.103446 2184 leader_election.cc:288] T 00000000000000000000000000000000 P 4939c7ba397e4ab48e64602dd2f1047c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 14:51:34.103538 2189 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4939c7ba397e4ab48e64602dd2f1047c [term 1 FOLLOWER]: Leader election won for term 1
I20260927 14:51:34.103749 2189 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4939c7ba397e4ab48e64602dd2f1047c [term 1 LEADER]: Becoming Leader. State: Replica: 4939c7ba397e4ab48e64602dd2f1047c, State: Running, Role: LEADER
I20260927 14:51:34.104038 2189 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4939c7ba397e4ab48e64602dd2f1047c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4939c7ba397e4ab48e64602dd2f1047c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40817 } }
I20260927 14:51:34.104797 2184 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4939c7ba397e4ab48e64602dd2f1047c [sys.catalog]: configured and running, proceeding with master startup.
I20260927 14:51:34.105031 2189 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4939c7ba397e4ab48e64602dd2f1047c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4939c7ba397e4ab48e64602dd2f1047c. Latest consensus state: current_term: 1 leader_uuid: "4939c7ba397e4ab48e64602dd2f1047c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4939c7ba397e4ab48e64602dd2f1047c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40817 } } }
I20260927 14:51:34.105147 2189 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4939c7ba397e4ab48e64602dd2f1047c [sys.catalog]: This master's current role is: LEADER
I20260927 14:51:34.105523 2193 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 14:51:34.105901 2190 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4939c7ba397e4ab48e64602dd2f1047c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4939c7ba397e4ab48e64602dd2f1047c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4939c7ba397e4ab48e64602dd2f1047c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40817 } } }
I20260927 14:51:34.106081 2190 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4939c7ba397e4ab48e64602dd2f1047c [sys.catalog]: This master's current role is: LEADER
I20260927 14:51:34.108551 2193 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 14:51:34.114223 2193 catalog_manager.cc:1347] Generated new cluster ID: 8100c1bb6b7a4e5e852f0f105c63f23b
I20260927 14:51:34.114300 2193 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 14:51:34.131072 2193 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 14:51:34.132021 2193 catalog_manager.cc:1504] Loading token signing keys...
I20260927 14:51:34.138383 2193 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4939c7ba397e4ab48e64602dd2f1047c: Generated new TSK 0
I20260927 14:51:34.139158 2193 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790520684602251-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790520684602251-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260927 14:51:34.176290 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6936m5/build/debug/bin/kudu
/tmp/dist-test-task6936m5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790520684602251-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790520684602251-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790520684602251-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790520684602251-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40817
--keytab_file=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790520684602251-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41793
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790520684602251-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790520684602251-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790520684602251-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 14:51:34.307646 2210 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 14:51:34.307943 2210 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 14:51:34.308022 2210 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 14:51:34.312188 2210 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 14:51:34.312269 2210 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 14:51:34.312336 2210 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260927 14:51:34.312376 2210 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 14:51:34.312449 2210 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 14:51:34.317893 2210 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41793
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790520684602251-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790520684602251-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790520684602251-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790520684602251-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:40817
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2210
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790520684602251-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:15:17 UTC on ccycloud.quasar-ledxfe.root.comops.site
build id 4676134
I20260927 14:51:34.318982 2210 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 14:51:34.320003 2210 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 14:51:34.329914 2216 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 14:51:34.330005 2215 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 14:51:34.329914 2218 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 14:51:34.330183 2210 server_base.cc:1034] running on GCE node
I20260927 14:51:34.330687 2210 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 14:51:34.331379 2210 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 14:51:34.332536 2210 hybrid_clock.cc:648] HybridClock initialized: now 1790520694332508 us; error 53 us; skew 500 ppm
Sep 27 14:51:34 dist-test-slave-xk1g krb5kdc[2093](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790520694, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 14:51:34.335637 2210 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 14:51:34.336550 2210 webserver.cc:460] Webserver started at http://127.0.98.1:38883/ using document root <none> and password file <none>
I20260927 14:51:34.337167 2210 fs_manager.cc:359] Metadata directory not provided
I20260927 14:51:34.337237 2210 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 14:51:34.337419 2210 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 14:51:34.339501 2210 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790520684602251-392-0/minicluster-data/ts-0/data/instance:
uuid: "29daca91da71400b9bf7f441f06b045c"
format_stamp: "Formatted at 2026-09-27 14:51:34 on dist-test-slave-xk1g"
server_key: "7db1b794246d93d70facea213bc73fd6"
server_key_iv: "926a06143dbc7a35f958579591d1ea9e"
server_key_version: "encryptionkey@0"
I20260927 14:51:34.340049 2210 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790520684602251-392-0/minicluster-data/ts-0/wal/instance:
uuid: "29daca91da71400b9bf7f441f06b045c"
format_stamp: "Formatted at 2026-09-27 14:51:34 on dist-test-slave-xk1g"
server_key: "7db1b794246d93d70facea213bc73fd6"
server_key_iv: "926a06143dbc7a35f958579591d1ea9e"
server_key_version: "encryptionkey@0"
I20260927 14:51:34.344038 2210 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260927 14:51:34.346618 2225 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 14:51:34.347951 2210 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 14:51:34.348058 2210 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790520684602251-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790520684602251-392-0/minicluster-data/ts-0/wal
uuid: "29daca91da71400b9bf7f441f06b045c"
format_stamp: "Formatted at 2026-09-27 14:51:34 on dist-test-slave-xk1g"
server_key: "7db1b794246d93d70facea213bc73fd6"
server_key_iv: "926a06143dbc7a35f958579591d1ea9e"
server_key_version: "encryptionkey@0"
I20260927 14:51:34.348150 2210 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790520684602251-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790520684602251-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790520684602251-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 14:51:34.366423 2210 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 14:51:34.369783 2210 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 14:51:34.369948 2210 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 14:51:34.371310 2210 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 14:51:34.371362 2210 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 14:51:34.371433 2210 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 14:51:34.371475 2210 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 27 14:51:34 dist-test-slave-xk1g krb5kdc[2093](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790520694, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 14:51:34.382495 2210 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:35643
I20260927 14:51:34.382535 2340 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:35643 every 8 connection(s)
I20260927 14:51:34.383519 2210 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790520684602251-392-0/minicluster-data/ts-0/data/info.pb
I20260927 14:51:34.386471 2187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:51:34.372327 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:60939 (local address 127.0.98.62:40817)
0927 14:51:34.372461 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0927 14:51:34.372465 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 14:51:34.373638 (+ 1173us) server_negotiation.cc:407] Connection header received
0927 14:51:34.374895 (+ 1257us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:51:34.374899 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:51:34.374965 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:51:34.375070 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:51:34.376785 (+ 1715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:51:34.377515 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:51:34.379426 (+ 1911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:51:34.379664 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:51:34.379748 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:51:34.382833 (+ 3085us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:51:34.382836 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:51:34.382838 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 14:51:34.382862 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:51:34.384479 (+ 1617us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:51:34.384968 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:51:34.384972 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:51:34.384977 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:51:34.385095 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:51:34.385411 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:51:34.385416 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:51:34.385418 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:51:34.385607 (+ 189us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:51:34.385707 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0927 14:51:34.386173 (+ 466us) server_negotiation.cc:299] Negotiation successful
0927 14:51:34.386294 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260927 14:51:34.387204 2234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:51:34.372897 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40817 (local address 127.0.98.1:60939)
0927 14:51:34.373467 (+ 570us) negotiation.cc:107] Waiting for socket to connect
0927 14:51:34.373515 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0927 14:51:34.374647 (+ 1132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:51:34.375287 (+ 640us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:51:34.375299 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:51:34.375831 (+ 532us) client_negotiation.cc:189] Negotiated authn=SASL
0927 14:51:34.376614 (+ 783us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:51:34.376637 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:51:34.377678 (+ 1041us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:51:34.377683 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:51:34.379305 (+ 1622us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:51:34.379316 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:51:34.379831 (+ 515us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:51:34.379837 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:51:34.380060 (+ 223us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:51:34.380797 (+ 737us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 14:51:34.380816 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0927 14:51:34.382647 (+ 1831us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 14:51:34.384599 (+ 1952us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 14:51:34.384605 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 14:51:34.384619 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0927 14:51:34.384868 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 14:51:34.385179 (+ 311us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 14:51:34.385181 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 14:51:34.385183 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 14:51:34.385319 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 14:51:34.385705 (+ 386us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 14:51:34.385712 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 14:51:34.385947 (+ 235us) client_negotiation.cc:769] Sending connection context
0927 14:51:34.386133 (+ 186us) client_negotiation.cc:240] Negotiation successful
0927 14:51:34.386437 (+ 304us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":359,"thread_start_us":144,"threads_started":1}
I20260927 14:51:34.388476 2341 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40817
I20260927 14:51:34.388648 2341 heartbeater.cc:464] Registering TS with master...
I20260927 14:51:34.389231 2341 heartbeater.cc:511] Master 127.0.98.62:40817 requested a full tablet report, sending...
I20260927 14:51:34.390990 2138 ts_manager.cc:194] Registered new tserver with Master: 29daca91da71400b9bf7f441f06b045c (127.0.98.1:35643)
I20260927 14:51:34.392118 2138 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:60939
I20260927 14:51:34.393659 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6936m5/build/debug/bin/kudu as pid 2210
I20260927 14:51:34.393788 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790520684602251-392-0/minicluster-data/ts-0/wal/instance
I20260927 14:51:34.394351 392 external_mini_cluster.cc:1428] Setting key 579b9dbe0e47b9fd2586c00b11ed15fc
I20260927 14:51:34.397768 2187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:51:34.390608 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:53587 (local address 127.0.98.62:40817)
0927 14:51:34.390753 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0927 14:51:34.390758 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 14:51:34.390913 (+ 155us) server_negotiation.cc:407] Connection header received
0927 14:51:34.391042 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:51:34.391044 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:51:34.391096 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:51:34.391184 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:51:34.391615 (+ 431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:51:34.392125 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:51:34.393150 (+ 1025us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:51:34.393364 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:51:34.393505 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:51:34.394652 (+ 1147us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:51:34.394656 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:51:34.394658 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 14:51:34.394685 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:51:34.396106 (+ 1421us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:51:34.396581 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:51:34.396583 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:51:34.396585 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:51:34.396653 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:51:34.396929 (+ 276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:51:34.396931 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:51:34.396932 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:51:34.397136 (+ 204us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:51:34.397239 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0927 14:51:34.397497 (+ 258us) server_negotiation.cc:299] Negotiation successful
0927 14:51:34.397603 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260927 14:51:34.397802 2347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:51:34.390544 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40817 (local address 127.0.98.1:53587)
0927 14:51:34.390832 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0927 14:51:34.390848 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 14:51:34.390942 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:51:34.391206 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:51:34.391209 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:51:34.391407 (+ 198us) client_negotiation.cc:189] Negotiated authn=SASL
0927 14:51:34.391499 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:51:34.391505 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:51:34.392288 (+ 783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:51:34.392290 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:51:34.392952 (+ 662us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:51:34.392962 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:51:34.393506 (+ 544us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:51:34.393518 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:51:34.393617 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:51:34.394089 (+ 472us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 14:51:34.394103 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0927 14:51:34.394526 (+ 423us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 14:51:34.396213 (+ 1687us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 14:51:34.396220 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 14:51:34.396225 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 14:51:34.396491 (+ 266us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 14:51:34.396734 (+ 243us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 14:51:34.396740 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 14:51:34.396745 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 14:51:34.396845 (+ 100us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 14:51:34.397240 (+ 395us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 14:51:34.397244 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 14:51:34.397339 (+ 95us) client_negotiation.cc:769] Sending connection context
0927 14:51:34.397446 (+ 107us) client_negotiation.cc:240] Negotiation successful
0927 14:51:34.397603 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":210,"mutex_wait_us":5,"thread_start_us":119,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790520684602251-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790520684602251-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260927 14:51:34.420894 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6936m5/build/debug/bin/kudu
/tmp/dist-test-task6936m5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790520684602251-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790520684602251-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790520684602251-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790520684602251-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40817
--keytab_file=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790520684602251-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41793
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790520684602251-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790520684602251-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790520684602251-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 14:51:34.532699 2350 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 14:51:34.533032 2350 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 14:51:34.533124 2350 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 14:51:34.537257 2350 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 14:51:34.537343 2350 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 14:51:34.537410 2350 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260927 14:51:34.537449 2350 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 14:51:34.537516 2350 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 14:51:34.543009 2350 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41793
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790520684602251-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790520684602251-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790520684602251-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790520684602251-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:40817
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2350
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790520684602251-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:15:17 UTC on ccycloud.quasar-ledxfe.root.comops.site
build id 4676134
I20260927 14:51:34.544183 2350 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 14:51:34.545320 2350 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 14:51:34.555311 2355 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 14:51:34.555330 2356 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 14:51:34.555507 2350 server_base.cc:1034] running on GCE node
W20260927 14:51:34.555315 2358 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 14:51:34.556080 2350 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 14:51:34.556788 2350 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 14:51:34.558012 2350 hybrid_clock.cc:648] HybridClock initialized: now 1790520694558010 us; error 51 us; skew 500 ppm
Sep 27 14:51:34 dist-test-slave-xk1g krb5kdc[2093](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790520694, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 14:51:34.560770 2350 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 14:51:34.561724 2350 webserver.cc:460] Webserver started at http://127.0.98.2:33691/ using document root <none> and password file <none>
I20260927 14:51:34.562295 2350 fs_manager.cc:359] Metadata directory not provided
I20260927 14:51:34.562382 2350 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 14:51:34.562611 2350 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 14:51:34.564713 2350 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790520684602251-392-0/minicluster-data/ts-1/data/instance:
uuid: "be25711ef9c347959ab3b8f8ae136a07"
format_stamp: "Formatted at 2026-09-27 14:51:34 on dist-test-slave-xk1g"
server_key: "f5573cfdcce933de588765a8105f14f3"
server_key_iv: "12de7698cd72ead61abc805729feb4a0"
server_key_version: "encryptionkey@0"
I20260927 14:51:34.565299 2350 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790520684602251-392-0/minicluster-data/ts-1/wal/instance:
uuid: "be25711ef9c347959ab3b8f8ae136a07"
format_stamp: "Formatted at 2026-09-27 14:51:34 on dist-test-slave-xk1g"
server_key: "f5573cfdcce933de588765a8105f14f3"
server_key_iv: "12de7698cd72ead61abc805729feb4a0"
server_key_version: "encryptionkey@0"
I20260927 14:51:34.569118 2350 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260927 14:51:34.571605 2365 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 14:51:34.572965 2350 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 14:51:34.573139 2350 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790520684602251-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790520684602251-392-0/minicluster-data/ts-1/wal
uuid: "be25711ef9c347959ab3b8f8ae136a07"
format_stamp: "Formatted at 2026-09-27 14:51:34 on dist-test-slave-xk1g"
server_key: "f5573cfdcce933de588765a8105f14f3"
server_key_iv: "12de7698cd72ead61abc805729feb4a0"
server_key_version: "encryptionkey@0"
I20260927 14:51:34.573256 2350 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790520684602251-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790520684602251-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790520684602251-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 14:51:34.602325 2350 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 14:51:34.605954 2350 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 14:51:34.606161 2350 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 14:51:34.607879 2350 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 14:51:34.607956 2350 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 14:51:34.608027 2350 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 14:51:34.608073 2350 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 27 14:51:34 dist-test-slave-xk1g krb5kdc[2093](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790520694, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 14:51:34.619200 2350 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:35577
I20260927 14:51:34.619262 2480 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:35577 every 8 connection(s)
I20260927 14:51:34.620261 2350 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790520684602251-392-0/minicluster-data/ts-1/data/info.pb
I20260927 14:51:34.621775 2187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:51:34.608514 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:37377 (local address 127.0.98.62:40817)
0927 14:51:34.608661 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0927 14:51:34.608665 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 14:51:34.609622 (+ 957us) server_negotiation.cc:407] Connection header received
0927 14:51:34.610702 (+ 1080us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:51:34.610706 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:51:34.610763 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:51:34.610827 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:51:34.612637 (+ 1810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:51:34.613397 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:51:34.614566 (+ 1169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:51:34.614865 (+ 299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:51:34.614993 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:51:34.617941 (+ 2948us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:51:34.617945 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:51:34.617947 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 14:51:34.617976 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:51:34.619544 (+ 1568us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:51:34.620153 (+ 609us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:51:34.620156 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:51:34.620157 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:51:34.620238 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:51:34.620605 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:51:34.620626 (+ 21us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:51:34.620629 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:51:34.620880 (+ 251us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:51:34.620977 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0927 14:51:34.621550 (+ 573us) server_negotiation.cc:299] Negotiation successful
0927 14:51:34.621657 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260927 14:51:34.622567 2374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:51:34.608944 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40817 (local address 127.0.98.2:37377)
0927 14:51:34.609447 (+ 503us) negotiation.cc:107] Waiting for socket to connect
0927 14:51:34.609508 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0927 14:51:34.610441 (+ 933us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:51:34.611302 (+ 861us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:51:34.611309 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:51:34.611818 (+ 509us) client_negotiation.cc:189] Negotiated authn=SASL
0927 14:51:34.612494 (+ 676us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:51:34.612517 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:51:34.613530 (+ 1013us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:51:34.613533 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:51:34.614430 (+ 897us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:51:34.614440 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:51:34.615034 (+ 594us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:51:34.615038 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:51:34.615222 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:51:34.615853 (+ 631us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 14:51:34.615875 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0927 14:51:34.617773 (+ 1898us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 14:51:34.619648 (+ 1875us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 14:51:34.619654 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 14:51:34.619669 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0927 14:51:34.620019 (+ 350us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 14:51:34.620337 (+ 318us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 14:51:34.620340 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 14:51:34.620342 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 14:51:34.620503 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 14:51:34.621031 (+ 528us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 14:51:34.621038 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 14:51:34.621328 (+ 290us) client_negotiation.cc:769] Sending connection context
0927 14:51:34.621486 (+ 158us) client_negotiation.cc:240] Negotiation successful
0927 14:51:34.621735 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":298,"thread_start_us":176,"threads_started":1}
I20260927 14:51:34.623795 2481 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40817
I20260927 14:51:34.623932 2481 heartbeater.cc:464] Registering TS with master...
I20260927 14:51:34.624498 2481 heartbeater.cc:511] Master 127.0.98.62:40817 requested a full tablet report, sending...
I20260927 14:51:34.625660 2138 ts_manager.cc:194] Registered new tserver with Master: be25711ef9c347959ab3b8f8ae136a07 (127.0.98.2:35577)
I20260927 14:51:34.626405 2138 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:37377
I20260927 14:51:34.627441 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6936m5/build/debug/bin/kudu as pid 2350
I20260927 14:51:34.627537 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790520684602251-392-0/minicluster-data/ts-1/wal/instance
I20260927 14:51:34.627811 392 external_mini_cluster.cc:1428] Setting key df7d16d7e6c319f472ad4f823a753ed9
I20260927 14:51:34.633972 2487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:51:34.625899 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40817 (local address 127.0.98.2:56529)
0927 14:51:34.626223 (+ 324us) negotiation.cc:107] Waiting for socket to connect
0927 14:51:34.626234 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 14:51:34.626317 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:51:34.626555 (+ 238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:51:34.626558 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:51:34.626751 (+ 193us) client_negotiation.cc:189] Negotiated authn=SASL
0927 14:51:34.626842 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:51:34.626847 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:51:34.627658 (+ 811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:51:34.627661 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:51:34.628465 (+ 804us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:51:34.628478 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:51:34.629130 (+ 652us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:51:34.629136 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:51:34.629229 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:51:34.629768 (+ 539us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 14:51:34.629782 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0927 14:51:34.630290 (+ 508us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 14:51:34.632300 (+ 2010us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 14:51:34.632306 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 14:51:34.632310 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 14:51:34.632628 (+ 318us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 14:51:34.632891 (+ 263us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 14:51:34.632894 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 14:51:34.632896 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 14:51:34.633001 (+ 105us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 14:51:34.633439 (+ 438us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 14:51:34.633443 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 14:51:34.633564 (+ 121us) client_negotiation.cc:769] Sending connection context
0927 14:51:34.633674 (+ 110us) client_negotiation.cc:240] Negotiation successful
0927 14:51:34.633828 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":255,"thread_start_us":91,"threads_started":1}
I20260927 14:51:34.634138 2187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:51:34.625976 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:56529 (local address 127.0.98.62:40817)
0927 14:51:34.626086 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0927 14:51:34.626091 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 14:51:34.626306 (+ 215us) server_negotiation.cc:407] Connection header received
0927 14:51:34.626381 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:51:34.626384 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:51:34.626440 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:51:34.626526 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:51:34.626969 (+ 443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:51:34.627500 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:51:34.628627 (+ 1127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:51:34.628970 (+ 343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:51:34.629124 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:51:34.630437 (+ 1313us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:51:34.630441 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:51:34.630443 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 14:51:34.630470 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:51:34.632179 (+ 1709us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:51:34.632726 (+ 547us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:51:34.632729 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:51:34.632730 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:51:34.632804 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:51:34.633139 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:51:34.633142 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:51:34.633144 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:51:34.633345 (+ 201us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:51:34.633421 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0927 14:51:34.633755 (+ 334us) server_negotiation.cc:299] Negotiation successful
0927 14:51:34.633891 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790520684602251-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790520684602251-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260927 14:51:34.657388 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6936m5/build/debug/bin/kudu
/tmp/dist-test-task6936m5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790520684602251-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790520684602251-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790520684602251-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790520684602251-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40817
--keytab_file=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790520684602251-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41793
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790520684602251-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790520684602251-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790520684602251-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 14:51:34.769284 2490 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 14:51:34.769551 2490 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 14:51:34.769601 2490 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 14:51:34.773614 2490 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 14:51:34.773682 2490 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 14:51:34.773726 2490 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260927 14:51:34.773744 2490 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 14:51:34.773779 2490 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 14:51:34.779158 2490 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41793
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790520684602251-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790520684602251-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790520684602251-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790520684602251-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:40817
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2490
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790520684602251-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:15:17 UTC on ccycloud.quasar-ledxfe.root.comops.site
build id 4676134
I20260927 14:51:34.780181 2490 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 14:51:34.781174 2490 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 14:51:34.790588 2496 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 14:51:34.790589 2495 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 14:51:34.790589 2498 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 14:51:34.790733 2490 server_base.cc:1034] running on GCE node
I20260927 14:51:34.791337 2490 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 14:51:34.791965 2490 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 14:51:34.793123 2490 hybrid_clock.cc:648] HybridClock initialized: now 1790520694793089 us; error 46 us; skew 500 ppm
Sep 27 14:51:34 dist-test-slave-xk1g krb5kdc[2093](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790520694, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 14:51:34.795801 2490 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 14:51:34.796656 2490 webserver.cc:460] Webserver started at http://127.0.98.3:39733/ using document root <none> and password file <none>
I20260927 14:51:34.797242 2490 fs_manager.cc:359] Metadata directory not provided
I20260927 14:51:34.797302 2490 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 14:51:34.797528 2490 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 14:51:34.799492 2490 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790520684602251-392-0/minicluster-data/ts-2/data/instance:
uuid: "7c70adccb37d4405976799ecc8680509"
format_stamp: "Formatted at 2026-09-27 14:51:34 on dist-test-slave-xk1g"
server_key: "85887f9c0df0bf2520548d2b4bb3b64d"
server_key_iv: "a592807ec5833a901eaae62b6ddc44ec"
server_key_version: "encryptionkey@0"
I20260927 14:51:34.800048 2490 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790520684602251-392-0/minicluster-data/ts-2/wal/instance:
uuid: "7c70adccb37d4405976799ecc8680509"
format_stamp: "Formatted at 2026-09-27 14:51:34 on dist-test-slave-xk1g"
server_key: "85887f9c0df0bf2520548d2b4bb3b64d"
server_key_iv: "a592807ec5833a901eaae62b6ddc44ec"
server_key_version: "encryptionkey@0"
I20260927 14:51:34.803750 2490 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260927 14:51:34.806393 2505 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 14:51:34.807684 2490 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 14:51:34.807803 2490 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790520684602251-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790520684602251-392-0/minicluster-data/ts-2/wal
uuid: "7c70adccb37d4405976799ecc8680509"
format_stamp: "Formatted at 2026-09-27 14:51:34 on dist-test-slave-xk1g"
server_key: "85887f9c0df0bf2520548d2b4bb3b64d"
server_key_iv: "a592807ec5833a901eaae62b6ddc44ec"
server_key_version: "encryptionkey@0"
I20260927 14:51:34.807930 2490 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790520684602251-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790520684602251-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790520684602251-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 14:51:34.825209 2490 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 14:51:34.828338 2490 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 14:51:34.828549 2490 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 14:51:34.830063 2490 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 14:51:34.830134 2490 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 14:51:34.830199 2490 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 14:51:34.830243 2490 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 27 14:51:34 dist-test-slave-xk1g krb5kdc[2093](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790520694, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 14:51:34.841112 2490 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:45935
I20260927 14:51:34.841171 2620 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:45935 every 8 connection(s)
I20260927 14:51:34.842006 2490 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790520684602251-392-0/minicluster-data/ts-2/data/info.pb
I20260927 14:51:34.843483 2187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:51:34.830718 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:36149 (local address 127.0.98.62:40817)
0927 14:51:34.830870 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0927 14:51:34.830877 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 14:51:34.831800 (+ 923us) server_negotiation.cc:407] Connection header received
0927 14:51:34.833085 (+ 1285us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:51:34.833092 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:51:34.833160 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:51:34.833276 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:51:34.834883 (+ 1607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:51:34.835612 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:51:34.836763 (+ 1151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:51:34.836993 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:51:34.837123 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:51:34.839993 (+ 2870us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:51:34.839997 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:51:34.839999 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 14:51:34.840027 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:51:34.841545 (+ 1518us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:51:34.842023 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:51:34.842028 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:51:34.842030 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:51:34.842125 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:51:34.842450 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:51:34.842457 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:51:34.842461 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:51:34.842671 (+ 210us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:51:34.842753 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0927 14:51:34.843260 (+ 507us) server_negotiation.cc:299] Negotiation successful
0927 14:51:34.843360 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260927 14:51:34.843730 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6936m5/build/debug/bin/kudu as pid 2490
I20260927 14:51:34.843840 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790520684602251-392-0/minicluster-data/ts-2/wal/instance
I20260927 14:51:34.844135 392 external_mini_cluster.cc:1428] Setting key afa255b627da950f0a7ea70161999c67
I20260927 14:51:34.844460 2514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:51:34.831146 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40817 (local address 127.0.98.3:36149)
0927 14:51:34.831620 (+ 474us) negotiation.cc:107] Waiting for socket to connect
0927 14:51:34.831677 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0927 14:51:34.832814 (+ 1137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:51:34.833469 (+ 655us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:51:34.833482 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:51:34.834009 (+ 527us) client_negotiation.cc:189] Negotiated authn=SASL
0927 14:51:34.834697 (+ 688us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:51:34.834719 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:51:34.835730 (+ 1011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:51:34.835753 (+ 23us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:51:34.836636 (+ 883us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:51:34.836645 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:51:34.837173 (+ 528us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:51:34.837177 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:51:34.837370 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:51:34.838106 (+ 736us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 14:51:34.838129 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0927 14:51:34.839811 (+ 1682us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 14:51:34.841660 (+ 1849us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 14:51:34.841665 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 14:51:34.841680 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0927 14:51:34.841926 (+ 246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 14:51:34.842218 (+ 292us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 14:51:34.842221 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 14:51:34.842223 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 14:51:34.842351 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 14:51:34.842818 (+ 467us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 14:51:34.842829 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 14:51:34.843044 (+ 215us) client_negotiation.cc:769] Sending connection context
0927 14:51:34.843196 (+ 152us) client_negotiation.cc:240] Negotiation successful
0927 14:51:34.843427 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":246,"thread_start_us":105,"threads_started":1}
I20260927 14:51:34.845681 2621 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40817
I20260927 14:51:34.845821 2621 heartbeater.cc:464] Registering TS with master...
I20260927 14:51:34.846554 2621 heartbeater.cc:511] Master 127.0.98.62:40817 requested a full tablet report, sending...
I20260927 14:51:34.847556 2137 ts_manager.cc:194] Registered new tserver with Master: 7c70adccb37d4405976799ecc8680509 (127.0.98.3:45935)
I20260927 14:51:34.848136 2137 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:36149
I20260927 14:51:34.854846 2187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:51:34.848391 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:43881 (local address 127.0.98.62:40817)
0927 14:51:34.848502 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0927 14:51:34.848506 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 14:51:34.848523 (+ 17us) server_negotiation.cc:407] Connection header received
0927 14:51:34.848551 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:51:34.848552 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:51:34.848582 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:51:34.848661 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:51:34.849110 (+ 449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:51:34.849814 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:51:34.850616 (+ 802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:51:34.850814 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:51:34.850889 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:51:34.851865 (+ 976us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:51:34.851868 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:51:34.851871 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 14:51:34.851894 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:51:34.853289 (+ 1395us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:51:34.853691 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:51:34.853695 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:51:34.853697 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:51:34.853777 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:51:34.854053 (+ 276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:51:34.854056 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:51:34.854058 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:51:34.854236 (+ 178us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:51:34.854337 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0927 14:51:34.854584 (+ 247us) server_negotiation.cc:299] Negotiation successful
0927 14:51:34.854692 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260927 14:51:34.854852 2627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:51:34.847672 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40817 (local address 127.0.98.3:43881)
0927 14:51:34.847919 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0927 14:51:34.847933 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 14:51:34.848010 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:51:34.848685 (+ 675us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:51:34.848688 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:51:34.848864 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0927 14:51:34.848982 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:51:34.848988 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:51:34.849950 (+ 962us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:51:34.849952 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:51:34.850522 (+ 570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:51:34.850529 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:51:34.850903 (+ 374us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:51:34.850905 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:51:34.850989 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:51:34.851401 (+ 412us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 14:51:34.851412 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 14:51:34.851713 (+ 301us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 14:51:34.853391 (+ 1678us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 14:51:34.853396 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 14:51:34.853398 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 14:51:34.853592 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 14:51:34.853865 (+ 273us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 14:51:34.853869 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 14:51:34.853871 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 14:51:34.853959 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 14:51:34.854328 (+ 369us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 14:51:34.854331 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 14:51:34.854424 (+ 93us) client_negotiation.cc:769] Sending connection context
0927 14:51:34.854530 (+ 106us) client_negotiation.cc:240] Negotiation successful
0927 14:51:34.854670 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":169,"thread_start_us":100,"threads_started":1}
I20260927 14:51:34.857975 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 27 14:51:34 dist-test-slave-xk1g krb5kdc[2093](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790520694, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 27 14:51:34 dist-test-slave-xk1g krb5kdc[2093](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790520694, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 14:51:34.882779 2187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:51:34.874828 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50588 (local address 127.0.98.62:40817)
0927 14:51:34.874958 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0927 14:51:34.874962 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 14:51:34.874973 (+ 11us) server_negotiation.cc:407] Connection header received
0927 14:51:34.875011 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:51:34.875013 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:51:34.875049 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:51:34.875149 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:51:34.875797 (+ 648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:51:34.876313 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:51:34.877277 (+ 964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:51:34.877543 (+ 266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:51:34.877650 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:51:34.880016 (+ 2366us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:51:34.880020 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:51:34.880022 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 14:51:34.880047 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:51:34.881435 (+ 1388us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:51:34.881810 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:51:34.881813 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:51:34.881814 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:51:34.881884 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:51:34.882120 (+ 236us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:51:34.882122 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:51:34.882123 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:51:34.882287 (+ 164us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:51:34.882377 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0927 14:51:34.882572 (+ 195us) server_negotiation.cc:299] Negotiation successful
0927 14:51:34.882661 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260927 14:51:34.885882 2137 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50588:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 14:51:34.888880 2137 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 14:51:34.904075 2643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:51:34.898296 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59184 (local address 127.0.98.3:45935)
0927 14:51:34.898659 (+ 363us) server_negotiation.cc:206] Beginning negotiation
0927 14:51:34.898670 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 14:51:34.899140 (+ 470us) server_negotiation.cc:407] Connection header received
0927 14:51:34.899216 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:51:34.899220 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:51:34.899395 (+ 175us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:51:34.899550 (+ 155us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 14:51:34.899860 (+ 310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:51:34.900797 (+ 937us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:51:34.902473 (+ 1676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:51:34.903267 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:51:34.903390 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:51:34.903555 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0927 14:51:34.903842 (+ 287us) server_negotiation.cc:299] Negotiation successful
0927 14:51:34.903924 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":256,"thread_start_us":120,"threads_started":1}
I20260927 14:51:34.904064 2641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:51:34.897839 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41838 (local address 127.0.98.1:35643)
0927 14:51:34.898276 (+ 437us) server_negotiation.cc:206] Beginning negotiation
0927 14:51:34.898280 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 14:51:34.898757 (+ 477us) server_negotiation.cc:407] Connection header received
0927 14:51:34.899556 (+ 799us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:51:34.899561 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:51:34.899759 (+ 198us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:51:34.899969 (+ 210us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 14:51:34.900197 (+ 228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:51:34.901407 (+ 1210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:51:34.902820 (+ 1413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:51:34.903355 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:51:34.903619 (+ 264us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:51:34.903777 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0927 14:51:34.903831 (+ 54us) server_negotiation.cc:299] Negotiation successful
0927 14:51:34.903923 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":330,"thread_start_us":131,"threads_started":1}
I20260927 14:51:34.904044 2642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:51:34.897540 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:35643 (local address 127.0.0.1:41838)
0927 14:51:34.898675 (+ 1135us) negotiation.cc:107] Waiting for socket to connect
0927 14:51:34.898687 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 14:51:34.898817 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:51:34.899931 (+ 1114us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:51:34.899936 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:51:34.899960 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 14:51:34.900066 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:51:34.900073 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:51:34.901532 (+ 1459us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:51:34.901536 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:51:34.902680 (+ 1144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:51:34.902691 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:51:34.903480 (+ 789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:51:34.903483 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:51:34.903564 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:51:34.903636 (+ 72us) client_negotiation.cc:769] Sending connection context
0927 14:51:34.903715 (+ 79us) client_negotiation.cc:240] Negotiation successful
0927 14:51:34.903872 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1069,"mutex_wait_us":36,"thread_start_us":91,"threads_started":1}
I20260927 14:51:34.904357 2639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:51:34.897499 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53790 (local address 127.0.98.2:35577)
0927 14:51:34.898092 (+ 593us) server_negotiation.cc:206] Beginning negotiation
0927 14:51:34.898106 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 14:51:34.898678 (+ 572us) server_negotiation.cc:407] Connection header received
0927 14:51:34.898980 (+ 302us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:51:34.898984 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:51:34.899188 (+ 204us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:51:34.899350 (+ 162us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 14:51:34.899751 (+ 401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:51:34.900945 (+ 1194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:51:34.902752 (+ 1807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:51:34.903513 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:51:34.903642 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:51:34.903805 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0927 14:51:34.904100 (+ 295us) server_negotiation.cc:299] Negotiation successful
0927 14:51:34.904208 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":487,"thread_start_us":118,"threads_started":1}
I20260927 14:51:34.905180 2640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:51:34.898034 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:45935 (local address 127.0.0.1:59184)
0927 14:51:34.899052 (+ 1018us) negotiation.cc:107] Waiting for socket to connect
0927 14:51:34.899062 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 14:51:34.899127 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:51:34.899544 (+ 417us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:51:34.899547 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:51:34.899568 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 14:51:34.899730 (+ 162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:51:34.899737 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:51:34.900966 (+ 1229us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:51:34.900969 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:51:34.902286 (+ 1317us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:51:34.902296 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:51:34.903405 (+ 1109us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:51:34.903412 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:51:34.903538 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:51:34.903609 (+ 71us) client_negotiation.cc:769] Sending connection context
0927 14:51:34.903728 (+ 119us) client_negotiation.cc:240] Negotiation successful
0927 14:51:34.905030 (+ 1302us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":962,"mutex_wait_us":565,"thread_start_us":83,"threads_started":1}
I20260927 14:51:34.905422 2638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:51:34.897361 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:35577 (local address 127.0.0.1:53790)
0927 14:51:34.898505 (+ 1144us) negotiation.cc:107] Waiting for socket to connect
0927 14:51:34.898555 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0927 14:51:34.898843 (+ 288us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:51:34.899386 (+ 543us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:51:34.899400 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:51:34.899424 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 14:51:34.899607 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:51:34.899616 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:51:34.901100 (+ 1484us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:51:34.901104 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:51:34.902615 (+ 1511us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:51:34.902626 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:51:34.903863 (+ 1237us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:51:34.903866 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:51:34.903951 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:51:34.903971 (+ 20us) client_negotiation.cc:769] Sending connection context
0927 14:51:34.905004 (+ 1033us) client_negotiation.cc:240] Negotiation successful
0927 14:51:34.905143 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":456,"mutex_wait_us":125,"thread_start_us":58,"threads_started":1}
I20260927 14:51:34.907272 2275 tablet_service.cc:1467] Processing CreateTablet for tablet 3b097ed60c3e412ba37b26144ae43c1a (DEFAULT_TABLE table=test-table [id=661f64defa944b4faa9159ad0c90065c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 14:51:34.907667 2415 tablet_service.cc:1467] Processing CreateTablet for tablet 3b097ed60c3e412ba37b26144ae43c1a (DEFAULT_TABLE table=test-table [id=661f64defa944b4faa9159ad0c90065c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 14:51:34.907753 2555 tablet_service.cc:1467] Processing CreateTablet for tablet 3b097ed60c3e412ba37b26144ae43c1a (DEFAULT_TABLE table=test-table [id=661f64defa944b4faa9159ad0c90065c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 14:51:34.908412 2275 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3b097ed60c3e412ba37b26144ae43c1a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 14:51:34.908588 2415 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3b097ed60c3e412ba37b26144ae43c1a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 14:51:34.908666 2555 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3b097ed60c3e412ba37b26144ae43c1a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 14:51:34.915237 2644 tablet_bootstrap.cc:492] T 3b097ed60c3e412ba37b26144ae43c1a P be25711ef9c347959ab3b8f8ae136a07: Bootstrap starting.
I20260927 14:51:34.916539 2646 tablet_bootstrap.cc:492] T 3b097ed60c3e412ba37b26144ae43c1a P 7c70adccb37d4405976799ecc8680509: Bootstrap starting.
I20260927 14:51:34.917383 2645 tablet_bootstrap.cc:492] T 3b097ed60c3e412ba37b26144ae43c1a P 29daca91da71400b9bf7f441f06b045c: Bootstrap starting.
I20260927 14:51:34.917601 2644 tablet_bootstrap.cc:654] T 3b097ed60c3e412ba37b26144ae43c1a P be25711ef9c347959ab3b8f8ae136a07: Neither blocks nor log segments found. Creating new log.
I20260927 14:51:34.918439 2644 log.cc:824] T 3b097ed60c3e412ba37b26144ae43c1a P be25711ef9c347959ab3b8f8ae136a07: Log is configured to *not* fsync() on all Append() calls
I20260927 14:51:34.918550 2646 tablet_bootstrap.cc:654] T 3b097ed60c3e412ba37b26144ae43c1a P 7c70adccb37d4405976799ecc8680509: Neither blocks nor log segments found. Creating new log.
I20260927 14:51:34.919317 2645 tablet_bootstrap.cc:654] T 3b097ed60c3e412ba37b26144ae43c1a P 29daca91da71400b9bf7f441f06b045c: Neither blocks nor log segments found. Creating new log.
I20260927 14:51:34.919322 2646 log.cc:824] T 3b097ed60c3e412ba37b26144ae43c1a P 7c70adccb37d4405976799ecc8680509: Log is configured to *not* fsync() on all Append() calls
I20260927 14:51:34.920068 2645 log.cc:824] T 3b097ed60c3e412ba37b26144ae43c1a P 29daca91da71400b9bf7f441f06b045c: Log is configured to *not* fsync() on all Append() calls
I20260927 14:51:34.920462 2644 tablet_bootstrap.cc:492] T 3b097ed60c3e412ba37b26144ae43c1a P be25711ef9c347959ab3b8f8ae136a07: No bootstrap required, opened a new log
I20260927 14:51:34.920660 2644 ts_tablet_manager.cc:1397] T 3b097ed60c3e412ba37b26144ae43c1a P be25711ef9c347959ab3b8f8ae136a07: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260927 14:51:34.921278 2646 tablet_bootstrap.cc:492] T 3b097ed60c3e412ba37b26144ae43c1a P 7c70adccb37d4405976799ecc8680509: No bootstrap required, opened a new log
I20260927 14:51:34.921474 2646 ts_tablet_manager.cc:1397] T 3b097ed60c3e412ba37b26144ae43c1a P 7c70adccb37d4405976799ecc8680509: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260927 14:51:34.921994 2645 tablet_bootstrap.cc:492] T 3b097ed60c3e412ba37b26144ae43c1a P 29daca91da71400b9bf7f441f06b045c: No bootstrap required, opened a new log
I20260927 14:51:34.922164 2645 ts_tablet_manager.cc:1397] T 3b097ed60c3e412ba37b26144ae43c1a P 29daca91da71400b9bf7f441f06b045c: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260927 14:51:34.923830 2644 raft_consensus.cc:348] T 3b097ed60c3e412ba37b26144ae43c1a P be25711ef9c347959ab3b8f8ae136a07 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be25711ef9c347959ab3b8f8ae136a07" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35577 } } peers { permanent_uuid: "29daca91da71400b9bf7f441f06b045c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35643 } } peers { permanent_uuid: "7c70adccb37d4405976799ecc8680509" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45935 } }
I20260927 14:51:34.924120 2644 raft_consensus.cc:374] T 3b097ed60c3e412ba37b26144ae43c1a P be25711ef9c347959ab3b8f8ae136a07 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 14:51:34.924186 2644 raft_consensus.cc:729] T 3b097ed60c3e412ba37b26144ae43c1a P be25711ef9c347959ab3b8f8ae136a07 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: be25711ef9c347959ab3b8f8ae136a07, State: Initialized, Role: FOLLOWER
I20260927 14:51:34.924665 2644 consensus_queue.cc:260] T 3b097ed60c3e412ba37b26144ae43c1a P be25711ef9c347959ab3b8f8ae136a07 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be25711ef9c347959ab3b8f8ae136a07" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35577 } } peers { permanent_uuid: "29daca91da71400b9bf7f441f06b045c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35643 } } peers { permanent_uuid: "7c70adccb37d4405976799ecc8680509" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45935 } }
I20260927 14:51:34.925586 2644 ts_tablet_manager.cc:1428] T 3b097ed60c3e412ba37b26144ae43c1a P be25711ef9c347959ab3b8f8ae136a07: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260927 14:51:34.925593 2646 raft_consensus.cc:348] T 3b097ed60c3e412ba37b26144ae43c1a P 7c70adccb37d4405976799ecc8680509 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be25711ef9c347959ab3b8f8ae136a07" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35577 } } peers { permanent_uuid: "29daca91da71400b9bf7f441f06b045c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35643 } } peers { permanent_uuid: "7c70adccb37d4405976799ecc8680509" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45935 } }
I20260927 14:51:34.925843 2646 raft_consensus.cc:374] T 3b097ed60c3e412ba37b26144ae43c1a P 7c70adccb37d4405976799ecc8680509 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 14:51:34.925897 2646 raft_consensus.cc:729] T 3b097ed60c3e412ba37b26144ae43c1a P 7c70adccb37d4405976799ecc8680509 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7c70adccb37d4405976799ecc8680509, State: Initialized, Role: FOLLOWER
I20260927 14:51:34.925930 2481 heartbeater.cc:503] Master 127.0.98.62:40817 was elected leader, sending a full tablet report...
I20260927 14:51:34.926299 2646 consensus_queue.cc:260] T 3b097ed60c3e412ba37b26144ae43c1a P 7c70adccb37d4405976799ecc8680509 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be25711ef9c347959ab3b8f8ae136a07" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35577 } } peers { permanent_uuid: "29daca91da71400b9bf7f441f06b045c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35643 } } peers { permanent_uuid: "7c70adccb37d4405976799ecc8680509" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45935 } }
I20260927 14:51:34.926574 2645 raft_consensus.cc:348] T 3b097ed60c3e412ba37b26144ae43c1a P 29daca91da71400b9bf7f441f06b045c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be25711ef9c347959ab3b8f8ae136a07" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35577 } } peers { permanent_uuid: "29daca91da71400b9bf7f441f06b045c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35643 } } peers { permanent_uuid: "7c70adccb37d4405976799ecc8680509" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45935 } }
I20260927 14:51:34.926796 2645 raft_consensus.cc:374] T 3b097ed60c3e412ba37b26144ae43c1a P 29daca91da71400b9bf7f441f06b045c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 14:51:34.926846 2645 raft_consensus.cc:729] T 3b097ed60c3e412ba37b26144ae43c1a P 29daca91da71400b9bf7f441f06b045c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 29daca91da71400b9bf7f441f06b045c, State: Initialized, Role: FOLLOWER
I20260927 14:51:34.927266 2646 ts_tablet_manager.cc:1428] T 3b097ed60c3e412ba37b26144ae43c1a P 7c70adccb37d4405976799ecc8680509: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260927 14:51:34.927275 2645 consensus_queue.cc:260] T 3b097ed60c3e412ba37b26144ae43c1a P 29daca91da71400b9bf7f441f06b045c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be25711ef9c347959ab3b8f8ae136a07" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35577 } } peers { permanent_uuid: "29daca91da71400b9bf7f441f06b045c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35643 } } peers { permanent_uuid: "7c70adccb37d4405976799ecc8680509" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45935 } }
I20260927 14:51:34.927605 2621 heartbeater.cc:503] Master 127.0.98.62:40817 was elected leader, sending a full tablet report...
I20260927 14:51:34.927882 2341 heartbeater.cc:503] Master 127.0.98.62:40817 was elected leader, sending a full tablet report...
I20260927 14:51:34.928153 2645 ts_tablet_manager.cc:1428] T 3b097ed60c3e412ba37b26144ae43c1a P 29daca91da71400b9bf7f441f06b045c: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260927 14:51:35.004714 2651 raft_consensus.cc:482] T 3b097ed60c3e412ba37b26144ae43c1a P 7c70adccb37d4405976799ecc8680509 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 14:51:35.004977 2651 raft_consensus.cc:504] T 3b097ed60c3e412ba37b26144ae43c1a P 7c70adccb37d4405976799ecc8680509 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be25711ef9c347959ab3b8f8ae136a07" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35577 } } peers { permanent_uuid: "29daca91da71400b9bf7f441f06b045c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35643 } } peers { permanent_uuid: "7c70adccb37d4405976799ecc8680509" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45935 } }
I20260927 14:51:35.006197 2651 leader_election.cc:288] T 3b097ed60c3e412ba37b26144ae43c1a P 7c70adccb37d4405976799ecc8680509 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers be25711ef9c347959ab3b8f8ae136a07 (127.0.98.2:35577), 29daca91da71400b9bf7f441f06b045c (127.0.98.1:35643)
I20260927 14:51:35.011090 2514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:51:35.006440 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:35577 (local address 127.0.98.3:44227)
0927 14:51:35.006664 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0927 14:51:35.006681 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 14:51:35.006783 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:51:35.007091 (+ 308us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:51:35.007093 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:51:35.007107 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 14:51:35.007239 (+ 132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:51:35.007245 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:51:35.008304 (+ 1059us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:51:35.008309 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:51:35.009758 (+ 1449us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:51:35.009769 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:51:35.010590 (+ 821us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:51:35.010592 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:51:35.010733 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:51:35.010752 (+ 19us) client_negotiation.cc:769] Sending connection context
0927 14:51:35.010837 (+ 85us) client_negotiation.cc:240] Negotiation successful
0927 14:51:35.010915 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114,"mutex_wait_us":40}
I20260927 14:51:35.011102 2639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:51:35.006518 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:44227 (local address 127.0.98.2:35577)
0927 14:51:35.006661 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0927 14:51:35.006665 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 14:51:35.006751 (+ 86us) server_negotiation.cc:407] Connection header received
0927 14:51:35.006898 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:51:35.006901 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:51:35.006979 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:51:35.007068 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 14:51:35.007348 (+ 280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:51:35.008150 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:51:35.009892 (+ 1742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:51:35.010481 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:51:35.010591 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:51:35.010653 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0927 14:51:35.010884 (+ 231us) server_negotiation.cc:299] Negotiation successful
0927 14:51:35.010965 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260927 14:51:35.011286 2641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:51:35.006586 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:52549 (local address 127.0.98.1:35643)
0927 14:51:35.006781 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0927 14:51:35.006785 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 14:51:35.006946 (+ 161us) server_negotiation.cc:407] Connection header received
0927 14:51:35.007042 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:51:35.007047 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:51:35.007107 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:51:35.007181 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 14:51:35.007442 (+ 261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:51:35.008176 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:51:35.010115 (+ 1939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:51:35.010727 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:51:35.010843 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:51:35.010930 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0927 14:51:35.011062 (+ 132us) server_negotiation.cc:299] Negotiation successful
0927 14:51:35.011142 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":110}
I20260927 14:51:35.011629 2653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:51:35.006474 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:35643 (local address 127.0.98.3:52549)
0927 14:51:35.006839 (+ 365us) negotiation.cc:107] Waiting for socket to connect
0927 14:51:35.006853 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 14:51:35.006933 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:51:35.007205 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:51:35.007210 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:51:35.007222 (+ 12us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 14:51:35.007333 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:51:35.007341 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:51:35.008304 (+ 963us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:51:35.008309 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:51:35.009997 (+ 1688us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:51:35.010003 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:51:35.010835 (+ 832us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:51:35.010838 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:51:35.010947 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:51:35.010968 (+ 21us) client_negotiation.cc:769] Sending connection context
0927 14:51:35.011037 (+ 69us) client_negotiation.cc:240] Negotiation successful
0927 14:51:35.011101 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":298,"mutex_wait_us":54,"thread_start_us":139,"threads_started":1}
I20260927 14:51:35.011876 2435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b097ed60c3e412ba37b26144ae43c1a" candidate_uuid: "7c70adccb37d4405976799ecc8680509" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "be25711ef9c347959ab3b8f8ae136a07" is_pre_election: true
I20260927 14:51:35.012216 2435 raft_consensus.cc:2454] T 3b097ed60c3e412ba37b26144ae43c1a P be25711ef9c347959ab3b8f8ae136a07 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7c70adccb37d4405976799ecc8680509 in term 0.
I20260927 14:51:35.012346 2295 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b097ed60c3e412ba37b26144ae43c1a" candidate_uuid: "7c70adccb37d4405976799ecc8680509" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "29daca91da71400b9bf7f441f06b045c" is_pre_election: true
I20260927 14:51:35.012651 2295 raft_consensus.cc:2454] T 3b097ed60c3e412ba37b26144ae43c1a P 29daca91da71400b9bf7f441f06b045c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7c70adccb37d4405976799ecc8680509 in term 0.
I20260927 14:51:35.012691 2508 leader_election.cc:302] T 3b097ed60c3e412ba37b26144ae43c1a P 7c70adccb37d4405976799ecc8680509 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7c70adccb37d4405976799ecc8680509, be25711ef9c347959ab3b8f8ae136a07; no voters:
I20260927 14:51:35.013024 2651 raft_consensus.cc:2781] T 3b097ed60c3e412ba37b26144ae43c1a P 7c70adccb37d4405976799ecc8680509 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 14:51:35.013139 2651 raft_consensus.cc:482] T 3b097ed60c3e412ba37b26144ae43c1a P 7c70adccb37d4405976799ecc8680509 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 14:51:35.013190 2651 raft_consensus.cc:3037] T 3b097ed60c3e412ba37b26144ae43c1a P 7c70adccb37d4405976799ecc8680509 [term 0 FOLLOWER]: Advancing to term 1
I20260927 14:51:35.014206 2651 raft_consensus.cc:504] T 3b097ed60c3e412ba37b26144ae43c1a P 7c70adccb37d4405976799ecc8680509 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be25711ef9c347959ab3b8f8ae136a07" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35577 } } peers { permanent_uuid: "29daca91da71400b9bf7f441f06b045c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35643 } } peers { permanent_uuid: "7c70adccb37d4405976799ecc8680509" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45935 } }
I20260927 14:51:35.014525 2651 leader_election.cc:288] T 3b097ed60c3e412ba37b26144ae43c1a P 7c70adccb37d4405976799ecc8680509 [CANDIDATE]: Term 1 election: Requested vote from peers be25711ef9c347959ab3b8f8ae136a07 (127.0.98.2:35577), 29daca91da71400b9bf7f441f06b045c (127.0.98.1:35643)
I20260927 14:51:35.014845 2435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b097ed60c3e412ba37b26144ae43c1a" candidate_uuid: "7c70adccb37d4405976799ecc8680509" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "be25711ef9c347959ab3b8f8ae136a07"
I20260927 14:51:35.014892 2295 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b097ed60c3e412ba37b26144ae43c1a" candidate_uuid: "7c70adccb37d4405976799ecc8680509" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "29daca91da71400b9bf7f441f06b045c"
I20260927 14:51:35.014976 2435 raft_consensus.cc:3037] T 3b097ed60c3e412ba37b26144ae43c1a P be25711ef9c347959ab3b8f8ae136a07 [term 0 FOLLOWER]: Advancing to term 1
I20260927 14:51:35.014998 2295 raft_consensus.cc:3037] T 3b097ed60c3e412ba37b26144ae43c1a P 29daca91da71400b9bf7f441f06b045c [term 0 FOLLOWER]: Advancing to term 1
I20260927 14:51:35.016296 2295 raft_consensus.cc:2454] T 3b097ed60c3e412ba37b26144ae43c1a P 29daca91da71400b9bf7f441f06b045c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7c70adccb37d4405976799ecc8680509 in term 1.
I20260927 14:51:35.016603 2507 leader_election.cc:302] T 3b097ed60c3e412ba37b26144ae43c1a P 7c70adccb37d4405976799ecc8680509 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 29daca91da71400b9bf7f441f06b045c, 7c70adccb37d4405976799ecc8680509; no voters:
I20260927 14:51:35.016685 2435 raft_consensus.cc:2454] T 3b097ed60c3e412ba37b26144ae43c1a P be25711ef9c347959ab3b8f8ae136a07 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7c70adccb37d4405976799ecc8680509 in term 1.
I20260927 14:51:35.016784 2651 raft_consensus.cc:2781] T 3b097ed60c3e412ba37b26144ae43c1a P 7c70adccb37d4405976799ecc8680509 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 14:51:35.016989 2651 raft_consensus.cc:686] T 3b097ed60c3e412ba37b26144ae43c1a P 7c70adccb37d4405976799ecc8680509 [term 1 LEADER]: Becoming Leader. State: Replica: 7c70adccb37d4405976799ecc8680509, State: Running, Role: LEADER
I20260927 14:51:35.017373 2651 consensus_queue.cc:237] T 3b097ed60c3e412ba37b26144ae43c1a P 7c70adccb37d4405976799ecc8680509 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be25711ef9c347959ab3b8f8ae136a07" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35577 } } peers { permanent_uuid: "29daca91da71400b9bf7f441f06b045c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35643 } } peers { permanent_uuid: "7c70adccb37d4405976799ecc8680509" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45935 } }
I20260927 14:51:35.020596 2136 catalog_manager.cc:5495] T 3b097ed60c3e412ba37b26144ae43c1a P 7c70adccb37d4405976799ecc8680509 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7c70adccb37d4405976799ecc8680509 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "7c70adccb37d4405976799ecc8680509" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be25711ef9c347959ab3b8f8ae136a07" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35577 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "29daca91da71400b9bf7f441f06b045c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35643 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7c70adccb37d4405976799ecc8680509" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45935 } health_report { overall_health: HEALTHY } } }
Sep 27 14:51:35 dist-test-slave-xk1g krb5kdc[2093](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790520694, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 14:51:35.083266 2187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:51:35.074765 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50600 (local address 127.0.98.62:40817)
0927 14:51:35.074892 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0927 14:51:35.074899 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 14:51:35.075100 (+ 201us) server_negotiation.cc:407] Connection header received
0927 14:51:35.075253 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:51:35.075256 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:51:35.075294 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:51:35.075372 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:51:35.076012 (+ 640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:51:35.076592 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:51:35.077496 (+ 904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:51:35.077713 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:51:35.077816 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:51:35.079985 (+ 2169us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:51:35.079988 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:51:35.079990 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 14:51:35.080013 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:51:35.081483 (+ 1470us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:51:35.082049 (+ 566us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:51:35.082052 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:51:35.082053 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:51:35.082127 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:51:35.082390 (+ 263us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:51:35.082393 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:51:35.082394 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:51:35.082568 (+ 174us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:51:35.082641 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0927 14:51:35.082977 (+ 336us) server_negotiation.cc:299] Negotiation successful
0927 14:51:35.083116 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":33,"server-negotiator.queue_time_us":38}
I20260927 14:51:35.083278 2661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:51:35.074720 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40817 (local address 127.0.0.1:50600)
0927 14:51:35.075016 (+ 296us) negotiation.cc:107] Waiting for socket to connect
0927 14:51:35.075028 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 14:51:35.075159 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:51:35.075387 (+ 228us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:51:35.075390 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:51:35.075810 (+ 420us) client_negotiation.cc:189] Negotiated authn=SASL
0927 14:51:35.075898 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:51:35.075904 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:51:35.076694 (+ 790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:51:35.076700 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:51:35.077386 (+ 686us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:51:35.077396 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:51:35.077819 (+ 423us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:51:35.077822 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:51:35.077900 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:51:35.078440 (+ 540us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 14:51:35.078451 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 14:51:35.079850 (+ 1399us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 14:51:35.081638 (+ 1788us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 14:51:35.081645 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 14:51:35.081652 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0927 14:51:35.081945 (+ 293us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 14:51:35.082213 (+ 268us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 14:51:35.082216 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 14:51:35.082218 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 14:51:35.082307 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 14:51:35.082664 (+ 357us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 14:51:35.082670 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 14:51:35.082788 (+ 118us) client_negotiation.cc:769] Sending connection context
0927 14:51:35.082910 (+ 122us) client_negotiation.cc:240] Negotiation successful
0927 14:51:35.083054 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":230,"thread_start_us":126,"threads_started":1}
I20260927 14:51:35.087069 2136 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:50600:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260927 14:51:35.087904 2136 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20260927 14:51:35.092973 2622 tablet.cc:2369] T 3b097ed60c3e412ba37b26144ae43c1a P 7c70adccb37d4405976799ecc8680509: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 14:51:35.092921 2275 tablet_service.cc:1467] Processing CreateTablet for tablet 6f20ed9c9eb34ffd98e0662a6677125b (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=a3027e2486f54b4183c49fc41792ca4b]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 14:51:35.092733 2415 tablet_service.cc:1467] Processing CreateTablet for tablet 6f20ed9c9eb34ffd98e0662a6677125b (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=a3027e2486f54b4183c49fc41792ca4b]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 14:51:35.092917 2555 tablet_service.cc:1467] Processing CreateTablet for tablet 6f20ed9c9eb34ffd98e0662a6677125b (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=a3027e2486f54b4183c49fc41792ca4b]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 14:51:35.093580 2555 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6f20ed9c9eb34ffd98e0662a6677125b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 14:51:35.093580 2275 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6f20ed9c9eb34ffd98e0662a6677125b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 14:51:35.093580 2415 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6f20ed9c9eb34ffd98e0662a6677125b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 14:51:35.096390 2646 tablet_bootstrap.cc:492] T 6f20ed9c9eb34ffd98e0662a6677125b P 7c70adccb37d4405976799ecc8680509: Bootstrap starting.
I20260927 14:51:35.096386 2645 tablet_bootstrap.cc:492] T 6f20ed9c9eb34ffd98e0662a6677125b P 29daca91da71400b9bf7f441f06b045c: Bootstrap starting.
I20260927 14:51:35.097431 2646 tablet_bootstrap.cc:654] T 6f20ed9c9eb34ffd98e0662a6677125b P 7c70adccb37d4405976799ecc8680509: Neither blocks nor log segments found. Creating new log.
I20260927 14:51:35.097438 2645 tablet_bootstrap.cc:654] T 6f20ed9c9eb34ffd98e0662a6677125b P 29daca91da71400b9bf7f441f06b045c: Neither blocks nor log segments found. Creating new log.
I20260927 14:51:35.097646 2644 tablet_bootstrap.cc:492] T 6f20ed9c9eb34ffd98e0662a6677125b P be25711ef9c347959ab3b8f8ae136a07: Bootstrap starting.
I20260927 14:51:35.098603 2644 tablet_bootstrap.cc:654] T 6f20ed9c9eb34ffd98e0662a6677125b P be25711ef9c347959ab3b8f8ae136a07: Neither blocks nor log segments found. Creating new log.
I20260927 14:51:35.098783 2645 tablet_bootstrap.cc:492] T 6f20ed9c9eb34ffd98e0662a6677125b P 29daca91da71400b9bf7f441f06b045c: No bootstrap required, opened a new log
I20260927 14:51:35.098783 2646 tablet_bootstrap.cc:492] T 6f20ed9c9eb34ffd98e0662a6677125b P 7c70adccb37d4405976799ecc8680509: No bootstrap required, opened a new log
I20260927 14:51:35.098868 2646 ts_tablet_manager.cc:1397] T 6f20ed9c9eb34ffd98e0662a6677125b P 7c70adccb37d4405976799ecc8680509: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260927 14:51:35.098868 2645 ts_tablet_manager.cc:1397] T 6f20ed9c9eb34ffd98e0662a6677125b P 29daca91da71400b9bf7f441f06b045c: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260927 14:51:35.099327 2646 raft_consensus.cc:348] T 6f20ed9c9eb34ffd98e0662a6677125b P 7c70adccb37d4405976799ecc8680509 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29daca91da71400b9bf7f441f06b045c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35643 } } peers { permanent_uuid: "be25711ef9c347959ab3b8f8ae136a07" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35577 } } peers { permanent_uuid: "7c70adccb37d4405976799ecc8680509" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45935 } }
I20260927 14:51:35.099327 2645 raft_consensus.cc:348] T 6f20ed9c9eb34ffd98e0662a6677125b P 29daca91da71400b9bf7f441f06b045c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29daca91da71400b9bf7f441f06b045c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35643 } } peers { permanent_uuid: "be25711ef9c347959ab3b8f8ae136a07" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35577 } } peers { permanent_uuid: "7c70adccb37d4405976799ecc8680509" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45935 } }
I20260927 14:51:35.099443 2645 raft_consensus.cc:374] T 6f20ed9c9eb34ffd98e0662a6677125b P 29daca91da71400b9bf7f441f06b045c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 14:51:35.099443 2646 raft_consensus.cc:374] T 6f20ed9c9eb34ffd98e0662a6677125b P 7c70adccb37d4405976799ecc8680509 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 14:51:35.099480 2646 raft_consensus.cc:729] T 6f20ed9c9eb34ffd98e0662a6677125b P 7c70adccb37d4405976799ecc8680509 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7c70adccb37d4405976799ecc8680509, State: Initialized, Role: FOLLOWER
I20260927 14:51:35.099479 2645 raft_consensus.cc:729] T 6f20ed9c9eb34ffd98e0662a6677125b P 29daca91da71400b9bf7f441f06b045c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 29daca91da71400b9bf7f441f06b045c, State: Initialized, Role: FOLLOWER
I20260927 14:51:35.099598 2646 consensus_queue.cc:260] T 6f20ed9c9eb34ffd98e0662a6677125b P 7c70adccb37d4405976799ecc8680509 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29daca91da71400b9bf7f441f06b045c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35643 } } peers { permanent_uuid: "be25711ef9c347959ab3b8f8ae136a07" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35577 } } peers { permanent_uuid: "7c70adccb37d4405976799ecc8680509" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45935 } }
I20260927 14:51:35.099598 2645 consensus_queue.cc:260] T 6f20ed9c9eb34ffd98e0662a6677125b P 29daca91da71400b9bf7f441f06b045c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29daca91da71400b9bf7f441f06b045c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35643 } } peers { permanent_uuid: "be25711ef9c347959ab3b8f8ae136a07" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35577 } } peers { permanent_uuid: "7c70adccb37d4405976799ecc8680509" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45935 } }
I20260927 14:51:35.099751 2644 tablet_bootstrap.cc:492] T 6f20ed9c9eb34ffd98e0662a6677125b P be25711ef9c347959ab3b8f8ae136a07: No bootstrap required, opened a new log
I20260927 14:51:35.099836 2644 ts_tablet_manager.cc:1397] T 6f20ed9c9eb34ffd98e0662a6677125b P be25711ef9c347959ab3b8f8ae136a07: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260927 14:51:35.099848 2646 ts_tablet_manager.cc:1428] T 6f20ed9c9eb34ffd98e0662a6677125b P 7c70adccb37d4405976799ecc8680509: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260927 14:51:35.099848 2645 ts_tablet_manager.cc:1428] T 6f20ed9c9eb34ffd98e0662a6677125b P 29daca91da71400b9bf7f441f06b045c: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260927 14:51:35.100072 2651 tablet_replica.cc:439] T 6f20ed9c9eb34ffd98e0662a6677125b P 7c70adccb37d4405976799ecc8680509: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29daca91da71400b9bf7f441f06b045c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35643 } } peers { permanent_uuid: "be25711ef9c347959ab3b8f8ae136a07" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35577 } } peers { permanent_uuid: "7c70adccb37d4405976799ecc8680509" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45935 } } }
I20260927 14:51:35.100205 2651 tablet_replica.cc:442] T 6f20ed9c9eb34ffd98e0662a6677125b P 7c70adccb37d4405976799ecc8680509: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 14:51:35.100283 2644 raft_consensus.cc:348] T 6f20ed9c9eb34ffd98e0662a6677125b P be25711ef9c347959ab3b8f8ae136a07 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29daca91da71400b9bf7f441f06b045c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35643 } } peers { permanent_uuid: "be25711ef9c347959ab3b8f8ae136a07" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35577 } } peers { permanent_uuid: "7c70adccb37d4405976799ecc8680509" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45935 } }
I20260927 14:51:35.100382 2644 raft_consensus.cc:374] T 6f20ed9c9eb34ffd98e0662a6677125b P be25711ef9c347959ab3b8f8ae136a07 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 14:51:35.100416 2644 raft_consensus.cc:729] T 6f20ed9c9eb34ffd98e0662a6677125b P be25711ef9c347959ab3b8f8ae136a07 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: be25711ef9c347959ab3b8f8ae136a07, State: Initialized, Role: FOLLOWER
I20260927 14:51:35.100478 2652 tablet_replica.cc:439] T 6f20ed9c9eb34ffd98e0662a6677125b P 29daca91da71400b9bf7f441f06b045c: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29daca91da71400b9bf7f441f06b045c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35643 } } peers { permanent_uuid: "be25711ef9c347959ab3b8f8ae136a07" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35577 } } peers { permanent_uuid: "7c70adccb37d4405976799ecc8680509" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45935 } } }
I20260927 14:51:35.100533 2644 consensus_queue.cc:260] T 6f20ed9c9eb34ffd98e0662a6677125b P be25711ef9c347959ab3b8f8ae136a07 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29daca91da71400b9bf7f441f06b045c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35643 } } peers { permanent_uuid: "be25711ef9c347959ab3b8f8ae136a07" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35577 } } peers { permanent_uuid: "7c70adccb37d4405976799ecc8680509" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45935 } }
I20260927 14:51:35.100610 2652 tablet_replica.cc:442] T 6f20ed9c9eb34ffd98e0662a6677125b P 29daca91da71400b9bf7f441f06b045c: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 14:51:35.100811 2644 ts_tablet_manager.cc:1428] T 6f20ed9c9eb34ffd98e0662a6677125b P be25711ef9c347959ab3b8f8ae136a07: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260927 14:51:35.101243 2650 tablet_replica.cc:439] T 6f20ed9c9eb34ffd98e0662a6677125b P be25711ef9c347959ab3b8f8ae136a07: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29daca91da71400b9bf7f441f06b045c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35643 } } peers { permanent_uuid: "be25711ef9c347959ab3b8f8ae136a07" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35577 } } peers { permanent_uuid: "7c70adccb37d4405976799ecc8680509" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45935 } } }
I20260927 14:51:35.101387 2650 tablet_replica.cc:442] T 6f20ed9c9eb34ffd98e0662a6677125b P be25711ef9c347959ab3b8f8ae136a07: This TxnStatusTablet replica's current role is: FOLLOWER
W20260927 14:51:35.121472 2482 tablet.cc:2369] T 3b097ed60c3e412ba37b26144ae43c1a P be25711ef9c347959ab3b8f8ae136a07: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 14:51:35.134857 2342 tablet.cc:2369] T 3b097ed60c3e412ba37b26144ae43c1a P 29daca91da71400b9bf7f441f06b045c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 14:51:35.309772 2652 raft_consensus.cc:482] T 6f20ed9c9eb34ffd98e0662a6677125b P 29daca91da71400b9bf7f441f06b045c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 14:51:35.309929 2652 raft_consensus.cc:504] T 6f20ed9c9eb34ffd98e0662a6677125b P 29daca91da71400b9bf7f441f06b045c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29daca91da71400b9bf7f441f06b045c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35643 } } peers { permanent_uuid: "be25711ef9c347959ab3b8f8ae136a07" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35577 } } peers { permanent_uuid: "7c70adccb37d4405976799ecc8680509" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45935 } }
I20260927 14:51:35.311331 2652 leader_election.cc:288] T 6f20ed9c9eb34ffd98e0662a6677125b P 29daca91da71400b9bf7f441f06b045c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers be25711ef9c347959ab3b8f8ae136a07 (127.0.98.2:35577), 7c70adccb37d4405976799ecc8680509 (127.0.98.3:45935)
I20260927 14:51:35.316422 2639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:51:35.311727 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:60623 (local address 127.0.98.2:35577)
0927 14:51:35.311851 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0927 14:51:35.311857 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 14:51:35.312162 (+ 305us) server_negotiation.cc:407] Connection header received
0927 14:51:35.312319 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:51:35.312322 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:51:35.312376 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:51:35.312475 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 14:51:35.313016 (+ 541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:51:35.313692 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:51:35.315199 (+ 1507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:51:35.315788 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:51:35.315897 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:51:35.315968 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0927 14:51:35.316200 (+ 232us) server_negotiation.cc:299] Negotiation successful
0927 14:51:35.316264 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260927 14:51:35.316489 2665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:51:35.311622 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:45935 (local address 127.0.98.1:52729)
0927 14:51:35.312144 (+ 522us) negotiation.cc:107] Waiting for socket to connect
0927 14:51:35.312157 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 14:51:35.312237 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:51:35.312526 (+ 289us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:51:35.312530 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:51:35.312578 (+ 48us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 14:51:35.313354 (+ 776us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:51:35.313363 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:51:35.314181 (+ 818us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:51:35.314185 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:51:35.315274 (+ 1089us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:51:35.315281 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:51:35.316101 (+ 820us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:51:35.316106 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:51:35.316209 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:51:35.316229 (+ 20us) client_negotiation.cc:769] Sending connection context
0927 14:51:35.316303 (+ 74us) client_negotiation.cc:240] Negotiation successful
0927 14:51:35.316367 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":459,"mutex_wait_us":122,"spinlock_wait_cycles":10624,"thread_start_us":107,"threads_started":1}
I20260927 14:51:35.316526 2643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:51:35.311697 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:52729 (local address 127.0.98.3:45935)
0927 14:51:35.311831 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0927 14:51:35.311836 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 14:51:35.312233 (+ 397us) server_negotiation.cc:407] Connection header received
0927 14:51:35.312317 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:51:35.312320 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:51:35.312376 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:51:35.312468 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 14:51:35.313488 (+ 1020us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:51:35.314065 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:51:35.315389 (+ 1324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:51:35.315993 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:51:35.316077 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:51:35.316121 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0927 14:51:35.316327 (+ 206us) server_negotiation.cc:299] Negotiation successful
0927 14:51:35.316396 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260927 14:51:35.316845 2666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:51:35.311622 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:35577 (local address 127.0.98.1:60623)
0927 14:51:35.312041 (+ 419us) negotiation.cc:107] Waiting for socket to connect
0927 14:51:35.312059 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 14:51:35.312198 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:51:35.312526 (+ 328us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:51:35.312530 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:51:35.312576 (+ 46us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 14:51:35.312858 (+ 282us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:51:35.312877 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:51:35.313798 (+ 921us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:51:35.313800 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:51:35.315055 (+ 1255us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:51:35.315066 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:51:35.315941 (+ 875us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:51:35.315944 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:51:35.316030 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:51:35.316082 (+ 52us) client_negotiation.cc:769] Sending connection context
0927 14:51:35.316171 (+ 89us) client_negotiation.cc:240] Negotiation successful
0927 14:51:35.316253 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":324,"mutex_wait_us":59,"spinlock_wait_cycles":28672,"thread_start_us":151,"threads_started":1}
I20260927 14:51:35.317240 2575 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f20ed9c9eb34ffd98e0662a6677125b" candidate_uuid: "29daca91da71400b9bf7f441f06b045c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7c70adccb37d4405976799ecc8680509" is_pre_election: true
I20260927 14:51:35.317422 2435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f20ed9c9eb34ffd98e0662a6677125b" candidate_uuid: "29daca91da71400b9bf7f441f06b045c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "be25711ef9c347959ab3b8f8ae136a07" is_pre_election: true
I20260927 14:51:35.317497 2575 raft_consensus.cc:2454] T 6f20ed9c9eb34ffd98e0662a6677125b P 7c70adccb37d4405976799ecc8680509 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 29daca91da71400b9bf7f441f06b045c in term 0.
I20260927 14:51:35.317541 2435 raft_consensus.cc:2454] T 6f20ed9c9eb34ffd98e0662a6677125b P be25711ef9c347959ab3b8f8ae136a07 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 29daca91da71400b9bf7f441f06b045c in term 0.
I20260927 14:51:35.317955 2228 leader_election.cc:302] T 6f20ed9c9eb34ffd98e0662a6677125b P 29daca91da71400b9bf7f441f06b045c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 29daca91da71400b9bf7f441f06b045c, 7c70adccb37d4405976799ecc8680509, be25711ef9c347959ab3b8f8ae136a07; no voters:
I20260927 14:51:35.318249 2652 raft_consensus.cc:2781] T 6f20ed9c9eb34ffd98e0662a6677125b P 29daca91da71400b9bf7f441f06b045c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 14:51:35.318325 2652 raft_consensus.cc:482] T 6f20ed9c9eb34ffd98e0662a6677125b P 29daca91da71400b9bf7f441f06b045c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 14:51:35.318375 2652 raft_consensus.cc:3037] T 6f20ed9c9eb34ffd98e0662a6677125b P 29daca91da71400b9bf7f441f06b045c [term 0 FOLLOWER]: Advancing to term 1
I20260927 14:51:35.319443 2652 raft_consensus.cc:504] T 6f20ed9c9eb34ffd98e0662a6677125b P 29daca91da71400b9bf7f441f06b045c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29daca91da71400b9bf7f441f06b045c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35643 } } peers { permanent_uuid: "be25711ef9c347959ab3b8f8ae136a07" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35577 } } peers { permanent_uuid: "7c70adccb37d4405976799ecc8680509" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45935 } }
I20260927 14:51:35.319855 2652 leader_election.cc:288] T 6f20ed9c9eb34ffd98e0662a6677125b P 29daca91da71400b9bf7f441f06b045c [CANDIDATE]: Term 1 election: Requested vote from peers be25711ef9c347959ab3b8f8ae136a07 (127.0.98.2:35577), 7c70adccb37d4405976799ecc8680509 (127.0.98.3:45935)
I20260927 14:51:35.320258 2435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f20ed9c9eb34ffd98e0662a6677125b" candidate_uuid: "29daca91da71400b9bf7f441f06b045c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "be25711ef9c347959ab3b8f8ae136a07"
I20260927 14:51:35.320258 2575 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f20ed9c9eb34ffd98e0662a6677125b" candidate_uuid: "29daca91da71400b9bf7f441f06b045c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7c70adccb37d4405976799ecc8680509"
I20260927 14:51:35.320385 2575 raft_consensus.cc:3037] T 6f20ed9c9eb34ffd98e0662a6677125b P 7c70adccb37d4405976799ecc8680509 [term 0 FOLLOWER]: Advancing to term 1
I20260927 14:51:35.320385 2435 raft_consensus.cc:3037] T 6f20ed9c9eb34ffd98e0662a6677125b P be25711ef9c347959ab3b8f8ae136a07 [term 0 FOLLOWER]: Advancing to term 1
I20260927 14:51:35.321573 2435 raft_consensus.cc:2454] T 6f20ed9c9eb34ffd98e0662a6677125b P be25711ef9c347959ab3b8f8ae136a07 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 29daca91da71400b9bf7f441f06b045c in term 1.
I20260927 14:51:35.321573 2575 raft_consensus.cc:2454] T 6f20ed9c9eb34ffd98e0662a6677125b P 7c70adccb37d4405976799ecc8680509 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 29daca91da71400b9bf7f441f06b045c in term 1.
I20260927 14:51:35.321854 2227 leader_election.cc:302] T 6f20ed9c9eb34ffd98e0662a6677125b P 29daca91da71400b9bf7f441f06b045c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 29daca91da71400b9bf7f441f06b045c, 7c70adccb37d4405976799ecc8680509; no voters:
I20260927 14:51:35.322023 2652 raft_consensus.cc:2781] T 6f20ed9c9eb34ffd98e0662a6677125b P 29daca91da71400b9bf7f441f06b045c [term 1 FOLLOWER]: Leader election won for term 1
I20260927 14:51:35.322232 2652 raft_consensus.cc:686] T 6f20ed9c9eb34ffd98e0662a6677125b P 29daca91da71400b9bf7f441f06b045c [term 1 LEADER]: Becoming Leader. State: Replica: 29daca91da71400b9bf7f441f06b045c, State: Running, Role: LEADER
I20260927 14:51:35.322598 2652 consensus_queue.cc:237] T 6f20ed9c9eb34ffd98e0662a6677125b P 29daca91da71400b9bf7f441f06b045c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29daca91da71400b9bf7f441f06b045c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35643 } } peers { permanent_uuid: "be25711ef9c347959ab3b8f8ae136a07" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35577 } } peers { permanent_uuid: "7c70adccb37d4405976799ecc8680509" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45935 } }
I20260927 14:51:35.324892 2667 tablet_replica.cc:439] T 6f20ed9c9eb34ffd98e0662a6677125b P 29daca91da71400b9bf7f441f06b045c: TxnStatusTablet state changed. Reason: New leader 29daca91da71400b9bf7f441f06b045c. Latest consensus state: current_term: 1 leader_uuid: "29daca91da71400b9bf7f441f06b045c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29daca91da71400b9bf7f441f06b045c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35643 } } peers { permanent_uuid: "be25711ef9c347959ab3b8f8ae136a07" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35577 } } peers { permanent_uuid: "7c70adccb37d4405976799ecc8680509" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45935 } } }
I20260927 14:51:35.324991 2667 tablet_replica.cc:442] T 6f20ed9c9eb34ffd98e0662a6677125b P 29daca91da71400b9bf7f441f06b045c: This TxnStatusTablet replica's current role is: LEADER
I20260927 14:51:35.325297 2669 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260927 14:51:35.325511 2669 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260927 14:51:35.325949 2138 catalog_manager.cc:5495] T 6f20ed9c9eb34ffd98e0662a6677125b P 29daca91da71400b9bf7f441f06b045c reported cstate change: term changed from 0 to 1, leader changed from <none> to 29daca91da71400b9bf7f441f06b045c (127.0.98.1). New cstate: current_term: 1 leader_uuid: "29daca91da71400b9bf7f441f06b045c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29daca91da71400b9bf7f441f06b045c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35643 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "be25711ef9c347959ab3b8f8ae136a07" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35577 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7c70adccb37d4405976799ecc8680509" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45935 } health_report { overall_health: UNKNOWN } } }
I20260927 14:51:35.341693 2641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:51:35.337837 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41848 (local address 127.0.98.1:35643)
0927 14:51:35.337990 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0927 14:51:35.337993 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 14:51:35.338007 (+ 14us) server_negotiation.cc:407] Connection header received
0927 14:51:35.338173 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:51:35.338175 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:51:35.338225 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:51:35.338297 (+ 72us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 14:51:35.338561 (+ 264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:51:35.339059 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:51:35.340250 (+ 1191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:51:35.340472 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:51:35.340564 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:51:35.340860 (+ 296us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 14:51:35.341326 (+ 466us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 14:51:35.341411 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0927 14:51:35.341521 (+ 110us) server_negotiation.cc:299] Negotiation successful
0927 14:51:35.341568 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260927 14:51:35.341733 2661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:51:35.337712 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:35643 (local address 127.0.0.1:41848)
0927 14:51:35.337909 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0927 14:51:35.337926 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 14:51:35.338029 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:51:35.338327 (+ 298us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:51:35.338331 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:51:35.338338 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 14:51:35.338460 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:51:35.338466 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:51:35.339155 (+ 689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:51:35.339159 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:51:35.340119 (+ 960us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:51:35.340130 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:51:35.340582 (+ 452us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:51:35.340585 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:51:35.340685 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:51:35.340736 (+ 51us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 14:51:35.341408 (+ 672us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 14:51:35.341422 (+ 14us) client_negotiation.cc:769] Sending connection context
0927 14:51:35.341504 (+ 82us) client_negotiation.cc:240] Negotiation successful
0927 14:51:35.341587 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":70}
I20260927 14:51:35.346804 2575 raft_consensus.cc:1261] T 6f20ed9c9eb34ffd98e0662a6677125b P 7c70adccb37d4405976799ecc8680509 [term 1 FOLLOWER]: Refusing update from remote peer 29daca91da71400b9bf7f441f06b045c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 14:51:35.346877 2435 raft_consensus.cc:1261] T 6f20ed9c9eb34ffd98e0662a6677125b P be25711ef9c347959ab3b8f8ae136a07 [term 1 FOLLOWER]: Refusing update from remote peer 29daca91da71400b9bf7f441f06b045c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 14:51:35.347381 2667 consensus_queue.cc:1035] T 6f20ed9c9eb34ffd98e0662a6677125b P 29daca91da71400b9bf7f441f06b045c [LEADER]: Connected to new peer: Peer: permanent_uuid: "7c70adccb37d4405976799ecc8680509" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 14:51:35.347574 2652 consensus_queue.cc:1035] T 6f20ed9c9eb34ffd98e0662a6677125b P 29daca91da71400b9bf7f441f06b045c [LEADER]: Connected to new peer: Peer: permanent_uuid: "be25711ef9c347959ab3b8f8ae136a07" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35577 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 14:51:35.351902 2650 tablet_replica.cc:439] T 6f20ed9c9eb34ffd98e0662a6677125b P be25711ef9c347959ab3b8f8ae136a07: TxnStatusTablet state changed. Reason: New leader 29daca91da71400b9bf7f441f06b045c. Latest consensus state: current_term: 1 leader_uuid: "29daca91da71400b9bf7f441f06b045c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29daca91da71400b9bf7f441f06b045c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35643 } } peers { permanent_uuid: "be25711ef9c347959ab3b8f8ae136a07" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35577 } } peers { permanent_uuid: "7c70adccb37d4405976799ecc8680509" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45935 } } }
I20260927 14:51:35.352034 2650 tablet_replica.cc:442] T 6f20ed9c9eb34ffd98e0662a6677125b P be25711ef9c347959ab3b8f8ae136a07: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 14:51:35.352393 2651 tablet_replica.cc:439] T 6f20ed9c9eb34ffd98e0662a6677125b P 7c70adccb37d4405976799ecc8680509: TxnStatusTablet state changed. Reason: New leader 29daca91da71400b9bf7f441f06b045c. Latest consensus state: current_term: 1 leader_uuid: "29daca91da71400b9bf7f441f06b045c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29daca91da71400b9bf7f441f06b045c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35643 } } peers { permanent_uuid: "be25711ef9c347959ab3b8f8ae136a07" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35577 } } peers { permanent_uuid: "7c70adccb37d4405976799ecc8680509" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45935 } } }
I20260927 14:51:35.352510 2651 tablet_replica.cc:442] T 6f20ed9c9eb34ffd98e0662a6677125b P 7c70adccb37d4405976799ecc8680509: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 14:51:35.355065 2652 tablet_replica.cc:439] T 6f20ed9c9eb34ffd98e0662a6677125b P 29daca91da71400b9bf7f441f06b045c: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "29daca91da71400b9bf7f441f06b045c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29daca91da71400b9bf7f441f06b045c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35643 } } peers { permanent_uuid: "be25711ef9c347959ab3b8f8ae136a07" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35577 } } peers { permanent_uuid: "7c70adccb37d4405976799ecc8680509" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45935 } } }
I20260927 14:51:35.355185 2652 tablet_replica.cc:442] T 6f20ed9c9eb34ffd98e0662a6677125b P 29daca91da71400b9bf7f441f06b045c: This TxnStatusTablet replica's current role is: LEADER
I20260927 14:51:35.355435 2651 tablet_replica.cc:439] T 6f20ed9c9eb34ffd98e0662a6677125b P 7c70adccb37d4405976799ecc8680509: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "29daca91da71400b9bf7f441f06b045c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29daca91da71400b9bf7f441f06b045c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35643 } } peers { permanent_uuid: "be25711ef9c347959ab3b8f8ae136a07" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35577 } } peers { permanent_uuid: "7c70adccb37d4405976799ecc8680509" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45935 } } }
I20260927 14:51:35.355567 2651 tablet_replica.cc:442] T 6f20ed9c9eb34ffd98e0662a6677125b P 7c70adccb37d4405976799ecc8680509: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 14:51:35.356630 2650 tablet_replica.cc:439] T 6f20ed9c9eb34ffd98e0662a6677125b P be25711ef9c347959ab3b8f8ae136a07: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "29daca91da71400b9bf7f441f06b045c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29daca91da71400b9bf7f441f06b045c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35643 } } peers { permanent_uuid: "be25711ef9c347959ab3b8f8ae136a07" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35577 } } peers { permanent_uuid: "7c70adccb37d4405976799ecc8680509" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45935 } } }
I20260927 14:51:35.356737 2650 tablet_replica.cc:442] T 6f20ed9c9eb34ffd98e0662a6677125b P be25711ef9c347959ab3b8f8ae136a07: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 14:51:35.356778 2672 mvcc.cc:205] Tried to move back new op lower bound from 7333972768133648384 to 7333972768043995136. Current Snapshot: MvccSnapshot[applied={T|T < 7333972768133648384}]
I20260927 14:51:35.357600 2652 tablet_replica.cc:439] T 6f20ed9c9eb34ffd98e0662a6677125b P 29daca91da71400b9bf7f441f06b045c: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "29daca91da71400b9bf7f441f06b045c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29daca91da71400b9bf7f441f06b045c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35643 } } peers { permanent_uuid: "be25711ef9c347959ab3b8f8ae136a07" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35577 } } peers { permanent_uuid: "7c70adccb37d4405976799ecc8680509" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45935 } } }
I20260927 14:51:35.357703 2652 tablet_replica.cc:442] T 6f20ed9c9eb34ffd98e0662a6677125b P 29daca91da71400b9bf7f441f06b045c: This TxnStatusTablet replica's current role is: LEADER
I20260927 14:51:35.358975 2675 mvcc.cc:205] Tried to move back new op lower bound from 7333972768133648384 to 7333972768043995136. Current Snapshot: MvccSnapshot[applied={T|T < 7333972768133648384 or (T in {7333972768133648384})}]
I20260927 14:51:35.360147 2673 mvcc.cc:205] Tried to move back new op lower bound from 7333972768133648384 to 7333972768043995136. Current Snapshot: MvccSnapshot[applied={T|T < 7333972768133648384}]
I20260927 14:51:35.369692 2643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:51:35.365561 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59190 (local address 127.0.98.3:45935)
0927 14:51:35.365676 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0927 14:51:35.365681 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 14:51:35.365700 (+ 19us) server_negotiation.cc:407] Connection header received
0927 14:51:35.365863 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:51:35.365866 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:51:35.365929 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:51:35.366039 (+ 110us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 14:51:35.366254 (+ 215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:51:35.367036 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:51:35.367912 (+ 876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:51:35.368260 (+ 348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:51:35.368363 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:51:35.368670 (+ 307us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 14:51:35.369265 (+ 595us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 14:51:35.369376 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0927 14:51:35.369487 (+ 111us) server_negotiation.cc:299] Negotiation successful
0927 14:51:35.369547 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260927 14:51:35.381033 2641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:51:35.377505 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:41929 (local address 127.0.98.1:35643)
0927 14:51:35.377621 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0927 14:51:35.377625 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 14:51:35.377847 (+ 222us) server_negotiation.cc:407] Connection header received
0927 14:51:35.377966 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:51:35.377968 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:51:35.378018 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:51:35.378085 (+ 67us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 14:51:35.378341 (+ 256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:51:35.378829 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:51:35.380027 (+ 1198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:51:35.380237 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:51:35.380309 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:51:35.380639 (+ 330us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 14:51:35.380700 (+ 61us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 14:51:35.380751 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0927 14:51:35.380861 (+ 110us) server_negotiation.cc:299] Negotiation successful
0927 14:51:35.380904 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260927 14:51:35.381119 2683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:51:35.377388 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:35643 (local address 127.0.98.3:41929)
0927 14:51:35.377761 (+ 373us) negotiation.cc:107] Waiting for socket to connect
0927 14:51:35.377775 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 14:51:35.377877 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:51:35.378108 (+ 231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:51:35.378111 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:51:35.378119 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 14:51:35.378237 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:51:35.378243 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:51:35.378925 (+ 682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:51:35.378928 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:51:35.379905 (+ 977us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:51:35.379915 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:51:35.380342 (+ 427us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:51:35.380346 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:51:35.380440 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:51:35.380498 (+ 58us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 14:51:35.380774 (+ 276us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 14:51:35.380780 (+ 6us) client_negotiation.cc:769] Sending connection context
0927 14:51:35.380847 (+ 67us) client_negotiation.cc:240] Negotiation successful
0927 14:51:35.380923 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":284,"thread_start_us":109,"threads_started":1}
I20260927 14:51:35.390462 2435 raft_consensus.cc:1261] T 3b097ed60c3e412ba37b26144ae43c1a P be25711ef9c347959ab3b8f8ae136a07 [term 1 FOLLOWER]: Refusing update from remote peer 7c70adccb37d4405976799ecc8680509: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 14:51:35.390621 2295 raft_consensus.cc:1261] T 3b097ed60c3e412ba37b26144ae43c1a P 29daca91da71400b9bf7f441f06b045c [term 1 FOLLOWER]: Refusing update from remote peer 7c70adccb37d4405976799ecc8680509: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 14:51:35.390957 2651 consensus_queue.cc:1035] T 3b097ed60c3e412ba37b26144ae43c1a P 7c70adccb37d4405976799ecc8680509 [LEADER]: Connected to new peer: Peer: permanent_uuid: "be25711ef9c347959ab3b8f8ae136a07" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35577 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 14:51:35.391237 2654 consensus_queue.cc:1035] T 3b097ed60c3e412ba37b26144ae43c1a P 7c70adccb37d4405976799ecc8680509 [LEADER]: Connected to new peer: Peer: permanent_uuid: "29daca91da71400b9bf7f441f06b045c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35643 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 14:51:35.416079 2643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:51:35.412385 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:50947 (local address 127.0.98.3:45935)
0927 14:51:35.412508 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0927 14:51:35.412513 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 14:51:35.412703 (+ 190us) server_negotiation.cc:407] Connection header received
0927 14:51:35.412789 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:51:35.412792 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:51:35.412839 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:51:35.412922 (+ 83us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 14:51:35.413157 (+ 235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:51:35.413892 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:51:35.415095 (+ 1203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:51:35.415303 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:51:35.415381 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:51:35.415684 (+ 303us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 14:51:35.415744 (+ 60us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 14:51:35.415796 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0927 14:51:35.415926 (+ 130us) server_negotiation.cc:299] Negotiation successful
0927 14:51:35.415972 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260927 14:51:35.416153 2690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:51:35.412318 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:45935 (local address 127.0.98.1:50947)
0927 14:51:35.412625 (+ 307us) negotiation.cc:107] Waiting for socket to connect
0927 14:51:35.412638 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 14:51:35.412722 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:51:35.412924 (+ 202us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:51:35.412926 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:51:35.412930 (+ 4us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 14:51:35.413032 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:51:35.413036 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:51:35.414021 (+ 985us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:51:35.414025 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:51:35.414983 (+ 958us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:51:35.414995 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:51:35.415416 (+ 421us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:51:35.415419 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:51:35.415515 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:51:35.415567 (+ 52us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 14:51:35.415825 (+ 258us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 14:51:35.415831 (+ 6us) client_negotiation.cc:769] Sending connection context
0927 14:51:35.415900 (+ 69us) client_negotiation.cc:240] Negotiation successful
0927 14:51:35.415978 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":113,"threads_started":1}
I20260927 14:51:35.441642 2136 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50588:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 14:51:35.441820 2136 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50588:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 14:51:35.444032 2136 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4939c7ba397e4ab48e64602dd2f1047c: Sending DeleteTablet for 3 replicas of tablet 3b097ed60c3e412ba37b26144ae43c1a
I20260927 14:51:35.444693 2415 tablet_service.cc:1514] Processing DeleteTablet for tablet 3b097ed60c3e412ba37b26144ae43c1a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 14:51:35 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:53790
I20260927 14:51:35.444921 2275 tablet_service.cc:1514] Processing DeleteTablet for tablet 3b097ed60c3e412ba37b26144ae43c1a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 14:51:35 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:41838
I20260927 14:51:35.445346 2694 tablet_replica.cc:330] T 3b097ed60c3e412ba37b26144ae43c1a P be25711ef9c347959ab3b8f8ae136a07: stopping tablet replica
I20260927 14:51:35.445456 2555 tablet_service.cc:1514] Processing DeleteTablet for tablet 3b097ed60c3e412ba37b26144ae43c1a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 14:51:35 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:59184
I20260927 14:51:35.445845 2694 raft_consensus.cc:2229] T 3b097ed60c3e412ba37b26144ae43c1a P be25711ef9c347959ab3b8f8ae136a07 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 14:51:35.446108 2694 raft_consensus.cc:2258] T 3b097ed60c3e412ba37b26144ae43c1a P be25711ef9c347959ab3b8f8ae136a07 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 14:51:35.446682 2136 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50588:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 14:51:35.447104 2136 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 14:51:35.447950 2694 ts_tablet_manager.cc:1905] T 3b097ed60c3e412ba37b26144ae43c1a P be25711ef9c347959ab3b8f8ae136a07: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 14:51:35.448437 2695 tablet_replica.cc:330] T 3b097ed60c3e412ba37b26144ae43c1a P 29daca91da71400b9bf7f441f06b045c: stopping tablet replica
I20260927 14:51:35.448737 2695 raft_consensus.cc:2229] T 3b097ed60c3e412ba37b26144ae43c1a P 29daca91da71400b9bf7f441f06b045c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 14:51:35.448951 2695 raft_consensus.cc:2258] T 3b097ed60c3e412ba37b26144ae43c1a P 29daca91da71400b9bf7f441f06b045c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 14:51:35.451359 2695 ts_tablet_manager.cc:1905] T 3b097ed60c3e412ba37b26144ae43c1a P 29daca91da71400b9bf7f441f06b045c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 14:51:35.452320 2275 tablet_service.cc:1467] Processing CreateTablet for tablet 5cdbea9f22bc492cb0a10259e6b5319b (DEFAULT_TABLE table=test-table [id=8ec7161675cd492b89c8baa5fc700a40]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 14:51:35.452579 2275 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5cdbea9f22bc492cb0a10259e6b5319b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 14:51:35.452616 2415 tablet_service.cc:1467] Processing CreateTablet for tablet 5cdbea9f22bc492cb0a10259e6b5319b (DEFAULT_TABLE table=test-table [id=8ec7161675cd492b89c8baa5fc700a40]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 14:51:35.453230 2696 tablet_replica.cc:330] T 3b097ed60c3e412ba37b26144ae43c1a P 7c70adccb37d4405976799ecc8680509: stopping tablet replica
I20260927 14:51:35.453502 2696 raft_consensus.cc:2229] T 3b097ed60c3e412ba37b26144ae43c1a P 7c70adccb37d4405976799ecc8680509 [term 1 LEADER]: Raft consensus shutting down.
I20260927 14:51:35.453822 2696 raft_consensus.cc:2258] T 3b097ed60c3e412ba37b26144ae43c1a P 7c70adccb37d4405976799ecc8680509 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 14:51:35.454062 2415 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5cdbea9f22bc492cb0a10259e6b5319b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 14:51:35.455068 2645 tablet_bootstrap.cc:492] T 5cdbea9f22bc492cb0a10259e6b5319b P 29daca91da71400b9bf7f441f06b045c: Bootstrap starting.
I20260927 14:51:35.455268 2555 tablet_service.cc:1467] Processing CreateTablet for tablet 5cdbea9f22bc492cb0a10259e6b5319b (DEFAULT_TABLE table=test-table [id=8ec7161675cd492b89c8baa5fc700a40]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 14:51:35.455528 2555 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5cdbea9f22bc492cb0a10259e6b5319b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 14:51:35.456022 2645 tablet_bootstrap.cc:654] T 5cdbea9f22bc492cb0a10259e6b5319b P 29daca91da71400b9bf7f441f06b045c: Neither blocks nor log segments found. Creating new log.
I20260927 14:51:35.456532 2644 tablet_bootstrap.cc:492] T 5cdbea9f22bc492cb0a10259e6b5319b P be25711ef9c347959ab3b8f8ae136a07: Bootstrap starting.
I20260927 14:51:35.456566 2696 ts_tablet_manager.cc:1905] T 3b097ed60c3e412ba37b26144ae43c1a P 7c70adccb37d4405976799ecc8680509: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 14:51:35.457450 2644 tablet_bootstrap.cc:654] T 5cdbea9f22bc492cb0a10259e6b5319b P be25711ef9c347959ab3b8f8ae136a07: Neither blocks nor log segments found. Creating new log.
I20260927 14:51:35.458025 2646 tablet_bootstrap.cc:492] T 5cdbea9f22bc492cb0a10259e6b5319b P 7c70adccb37d4405976799ecc8680509: Bootstrap starting.
I20260927 14:51:35.458943 2646 tablet_bootstrap.cc:654] T 5cdbea9f22bc492cb0a10259e6b5319b P 7c70adccb37d4405976799ecc8680509: Neither blocks nor log segments found. Creating new log.
I20260927 14:51:35.459198 2696 ts_tablet_manager.cc:1918] T 3b097ed60c3e412ba37b26144ae43c1a P 7c70adccb37d4405976799ecc8680509: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 14:51:35.459290 2696 log.cc:1196] T 3b097ed60c3e412ba37b26144ae43c1a P 7c70adccb37d4405976799ecc8680509: Deleting WAL directory at /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790520684602251-392-0/minicluster-data/ts-2/wal/wals/3b097ed60c3e412ba37b26144ae43c1a
I20260927 14:51:35.459595 2696 ts_tablet_manager.cc:1939] T 3b097ed60c3e412ba37b26144ae43c1a P 7c70adccb37d4405976799ecc8680509: Deleting consensus metadata
I20260927 14:51:35.460842 2123 catalog_manager.cc:4841] TS 7c70adccb37d4405976799ecc8680509 (127.0.98.3:45935): tablet 3b097ed60c3e412ba37b26144ae43c1a (table test-table [id=661f64defa944b4faa9159ad0c90065c]) successfully deleted
I20260927 14:51:35.462621 2694 ts_tablet_manager.cc:1918] T 3b097ed60c3e412ba37b26144ae43c1a P be25711ef9c347959ab3b8f8ae136a07: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 14:51:35.462621 2695 ts_tablet_manager.cc:1918] T 3b097ed60c3e412ba37b26144ae43c1a P 29daca91da71400b9bf7f441f06b045c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 14:51:35.462718 2695 log.cc:1196] T 3b097ed60c3e412ba37b26144ae43c1a P 29daca91da71400b9bf7f441f06b045c: Deleting WAL directory at /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790520684602251-392-0/minicluster-data/ts-0/wal/wals/3b097ed60c3e412ba37b26144ae43c1a
I20260927 14:51:35.462868 2694 log.cc:1196] T 3b097ed60c3e412ba37b26144ae43c1a P be25711ef9c347959ab3b8f8ae136a07: Deleting WAL directory at /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790520684602251-392-0/minicluster-data/ts-1/wal/wals/3b097ed60c3e412ba37b26144ae43c1a
I20260927 14:51:35.463059 2695 ts_tablet_manager.cc:1939] T 3b097ed60c3e412ba37b26144ae43c1a P 29daca91da71400b9bf7f441f06b045c: Deleting consensus metadata
I20260927 14:51:35.463274 2694 ts_tablet_manager.cc:1939] T 3b097ed60c3e412ba37b26144ae43c1a P be25711ef9c347959ab3b8f8ae136a07: Deleting consensus metadata
I20260927 14:51:35.464633 2124 catalog_manager.cc:4841] TS be25711ef9c347959ab3b8f8ae136a07 (127.0.98.2:35577): tablet 3b097ed60c3e412ba37b26144ae43c1a (table test-table [id=661f64defa944b4faa9159ad0c90065c]) successfully deleted
I20260927 14:51:35.464950 2646 tablet_bootstrap.cc:492] T 5cdbea9f22bc492cb0a10259e6b5319b P 7c70adccb37d4405976799ecc8680509: No bootstrap required, opened a new log
I20260927 14:51:35.465040 2646 ts_tablet_manager.cc:1397] T 5cdbea9f22bc492cb0a10259e6b5319b P 7c70adccb37d4405976799ecc8680509: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.000s
I20260927 14:51:35.465548 2646 raft_consensus.cc:348] T 5cdbea9f22bc492cb0a10259e6b5319b P 7c70adccb37d4405976799ecc8680509 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29daca91da71400b9bf7f441f06b045c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35643 } } peers { permanent_uuid: "be25711ef9c347959ab3b8f8ae136a07" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35577 } } peers { permanent_uuid: "7c70adccb37d4405976799ecc8680509" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45935 } }
I20260927 14:51:35.465690 2646 raft_consensus.cc:374] T 5cdbea9f22bc492cb0a10259e6b5319b P 7c70adccb37d4405976799ecc8680509 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 14:51:35.465731 2646 raft_consensus.cc:729] T 5cdbea9f22bc492cb0a10259e6b5319b P 7c70adccb37d4405976799ecc8680509 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7c70adccb37d4405976799ecc8680509, State: Initialized, Role: FOLLOWER
I20260927 14:51:35.465852 2646 consensus_queue.cc:260] T 5cdbea9f22bc492cb0a10259e6b5319b P 7c70adccb37d4405976799ecc8680509 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29daca91da71400b9bf7f441f06b045c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35643 } } peers { permanent_uuid: "be25711ef9c347959ab3b8f8ae136a07" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35577 } } peers { permanent_uuid: "7c70adccb37d4405976799ecc8680509" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45935 } }
I20260927 14:51:35.466122 2646 ts_tablet_manager.cc:1428] T 5cdbea9f22bc492cb0a10259e6b5319b P 7c70adccb37d4405976799ecc8680509: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260927 14:51:35.466674 2644 tablet_bootstrap.cc:492] T 5cdbea9f22bc492cb0a10259e6b5319b P be25711ef9c347959ab3b8f8ae136a07: No bootstrap required, opened a new log
I20260927 14:51:35.466747 2644 ts_tablet_manager.cc:1397] T 5cdbea9f22bc492cb0a10259e6b5319b P be25711ef9c347959ab3b8f8ae136a07: Time spent bootstrapping tablet: real 0.010s user 0.001s sys 0.000s
I20260927 14:51:35.467173 2644 raft_consensus.cc:348] T 5cdbea9f22bc492cb0a10259e6b5319b P be25711ef9c347959ab3b8f8ae136a07 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29daca91da71400b9bf7f441f06b045c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35643 } } peers { permanent_uuid: "be25711ef9c347959ab3b8f8ae136a07" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35577 } } peers { permanent_uuid: "7c70adccb37d4405976799ecc8680509" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45935 } }
I20260927 14:51:35.467290 2644 raft_consensus.cc:374] T 5cdbea9f22bc492cb0a10259e6b5319b P be25711ef9c347959ab3b8f8ae136a07 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 14:51:35.467319 2644 raft_consensus.cc:729] T 5cdbea9f22bc492cb0a10259e6b5319b P be25711ef9c347959ab3b8f8ae136a07 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: be25711ef9c347959ab3b8f8ae136a07, State: Initialized, Role: FOLLOWER
I20260927 14:51:35.467432 2644 consensus_queue.cc:260] T 5cdbea9f22bc492cb0a10259e6b5319b P be25711ef9c347959ab3b8f8ae136a07 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29daca91da71400b9bf7f441f06b045c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35643 } } peers { permanent_uuid: "be25711ef9c347959ab3b8f8ae136a07" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35577 } } peers { permanent_uuid: "7c70adccb37d4405976799ecc8680509" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45935 } }
I20260927 14:51:35.467729 2644 ts_tablet_manager.cc:1428] T 5cdbea9f22bc492cb0a10259e6b5319b P be25711ef9c347959ab3b8f8ae136a07: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260927 14:51:35.468901 2123 catalog_manager.cc:4841] TS 29daca91da71400b9bf7f441f06b045c (127.0.98.1:35643): tablet 3b097ed60c3e412ba37b26144ae43c1a (table test-table [id=661f64defa944b4faa9159ad0c90065c]) successfully deleted
I20260927 14:51:35.473274 2645 tablet_bootstrap.cc:492] T 5cdbea9f22bc492cb0a10259e6b5319b P 29daca91da71400b9bf7f441f06b045c: No bootstrap required, opened a new log
I20260927 14:51:35.473378 2645 ts_tablet_manager.cc:1397] T 5cdbea9f22bc492cb0a10259e6b5319b P 29daca91da71400b9bf7f441f06b045c: Time spent bootstrapping tablet: real 0.018s user 0.002s sys 0.000s
I20260927 14:51:35.473842 2645 raft_consensus.cc:348] T 5cdbea9f22bc492cb0a10259e6b5319b P 29daca91da71400b9bf7f441f06b045c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29daca91da71400b9bf7f441f06b045c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35643 } } peers { permanent_uuid: "be25711ef9c347959ab3b8f8ae136a07" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35577 } } peers { permanent_uuid: "7c70adccb37d4405976799ecc8680509" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45935 } }
I20260927 14:51:35.473963 2645 raft_consensus.cc:374] T 5cdbea9f22bc492cb0a10259e6b5319b P 29daca91da71400b9bf7f441f06b045c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 14:51:35.474004 2645 raft_consensus.cc:729] T 5cdbea9f22bc492cb0a10259e6b5319b P 29daca91da71400b9bf7f441f06b045c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 29daca91da71400b9bf7f441f06b045c, State: Initialized, Role: FOLLOWER
I20260927 14:51:35.474123 2645 consensus_queue.cc:260] T 5cdbea9f22bc492cb0a10259e6b5319b P 29daca91da71400b9bf7f441f06b045c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29daca91da71400b9bf7f441f06b045c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35643 } } peers { permanent_uuid: "be25711ef9c347959ab3b8f8ae136a07" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35577 } } peers { permanent_uuid: "7c70adccb37d4405976799ecc8680509" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45935 } }
I20260927 14:51:35.474399 2645 ts_tablet_manager.cc:1428] T 5cdbea9f22bc492cb0a10259e6b5319b P 29daca91da71400b9bf7f441f06b045c: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260927 14:51:35.600739 2650 raft_consensus.cc:482] T 5cdbea9f22bc492cb0a10259e6b5319b P be25711ef9c347959ab3b8f8ae136a07 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 14:51:35.600899 2650 raft_consensus.cc:504] T 5cdbea9f22bc492cb0a10259e6b5319b P be25711ef9c347959ab3b8f8ae136a07 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29daca91da71400b9bf7f441f06b045c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35643 } } peers { permanent_uuid: "be25711ef9c347959ab3b8f8ae136a07" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35577 } } peers { permanent_uuid: "7c70adccb37d4405976799ecc8680509" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45935 } }
I20260927 14:51:35.602239 2650 leader_election.cc:288] T 5cdbea9f22bc492cb0a10259e6b5319b P be25711ef9c347959ab3b8f8ae136a07 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 29daca91da71400b9bf7f441f06b045c (127.0.98.1:35643), 7c70adccb37d4405976799ecc8680509 (127.0.98.3:45935)
I20260927 14:51:35.602795 2692 raft_consensus.cc:482] T 5cdbea9f22bc492cb0a10259e6b5319b P 7c70adccb37d4405976799ecc8680509 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 14:51:35.602937 2692 raft_consensus.cc:504] T 5cdbea9f22bc492cb0a10259e6b5319b P 7c70adccb37d4405976799ecc8680509 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29daca91da71400b9bf7f441f06b045c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35643 } } peers { permanent_uuid: "be25711ef9c347959ab3b8f8ae136a07" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35577 } } peers { permanent_uuid: "7c70adccb37d4405976799ecc8680509" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45935 } }
I20260927 14:51:35.603309 2692 leader_election.cc:288] T 5cdbea9f22bc492cb0a10259e6b5319b P 7c70adccb37d4405976799ecc8680509 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 29daca91da71400b9bf7f441f06b045c (127.0.98.1:35643), be25711ef9c347959ab3b8f8ae136a07 (127.0.98.2:35577)
I20260927 14:51:35.603317 2667 raft_consensus.cc:482] T 5cdbea9f22bc492cb0a10259e6b5319b P 29daca91da71400b9bf7f441f06b045c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 14:51:35.603448 2667 raft_consensus.cc:504] T 5cdbea9f22bc492cb0a10259e6b5319b P 29daca91da71400b9bf7f441f06b045c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29daca91da71400b9bf7f441f06b045c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35643 } } peers { permanent_uuid: "be25711ef9c347959ab3b8f8ae136a07" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35577 } } peers { permanent_uuid: "7c70adccb37d4405976799ecc8680509" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45935 } }
I20260927 14:51:35.603820 2667 leader_election.cc:288] T 5cdbea9f22bc492cb0a10259e6b5319b P 29daca91da71400b9bf7f441f06b045c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers be25711ef9c347959ab3b8f8ae136a07 (127.0.98.2:35577), 7c70adccb37d4405976799ecc8680509 (127.0.98.3:45935)
I20260927 14:51:35.604106 2295 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5cdbea9f22bc492cb0a10259e6b5319b" candidate_uuid: "7c70adccb37d4405976799ecc8680509" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "29daca91da71400b9bf7f441f06b045c" is_pre_election: true
I20260927 14:51:35.604233 2295 raft_consensus.cc:2454] T 5cdbea9f22bc492cb0a10259e6b5319b P 29daca91da71400b9bf7f441f06b045c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7c70adccb37d4405976799ecc8680509 in term 0.
I20260927 14:51:35.604209 2435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5cdbea9f22bc492cb0a10259e6b5319b" candidate_uuid: "7c70adccb37d4405976799ecc8680509" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "be25711ef9c347959ab3b8f8ae136a07" is_pre_election: true
I20260927 14:51:35.604296 2434 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5cdbea9f22bc492cb0a10259e6b5319b" candidate_uuid: "29daca91da71400b9bf7f441f06b045c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "be25711ef9c347959ab3b8f8ae136a07" is_pre_election: true
I20260927 14:51:35.604352 2435 raft_consensus.cc:2454] T 5cdbea9f22bc492cb0a10259e6b5319b P be25711ef9c347959ab3b8f8ae136a07 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7c70adccb37d4405976799ecc8680509 in term 0.
I20260927 14:51:35.604857 2507 leader_election.cc:302] T 5cdbea9f22bc492cb0a10259e6b5319b P 7c70adccb37d4405976799ecc8680509 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 29daca91da71400b9bf7f441f06b045c, 7c70adccb37d4405976799ecc8680509; no voters:
I20260927 14:51:35.605046 2692 raft_consensus.cc:2781] T 5cdbea9f22bc492cb0a10259e6b5319b P 7c70adccb37d4405976799ecc8680509 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 14:51:35.605126 2692 raft_consensus.cc:482] T 5cdbea9f22bc492cb0a10259e6b5319b P 7c70adccb37d4405976799ecc8680509 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 14:51:35.605166 2692 raft_consensus.cc:3037] T 5cdbea9f22bc492cb0a10259e6b5319b P 7c70adccb37d4405976799ecc8680509 [term 0 FOLLOWER]: Advancing to term 1
I20260927 14:51:35.605289 2575 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5cdbea9f22bc492cb0a10259e6b5319b" candidate_uuid: "29daca91da71400b9bf7f441f06b045c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7c70adccb37d4405976799ecc8680509" is_pre_election: true
I20260927 14:51:35.606401 2692 raft_consensus.cc:504] T 5cdbea9f22bc492cb0a10259e6b5319b P 7c70adccb37d4405976799ecc8680509 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29daca91da71400b9bf7f441f06b045c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35643 } } peers { permanent_uuid: "be25711ef9c347959ab3b8f8ae136a07" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35577 } } peers { permanent_uuid: "7c70adccb37d4405976799ecc8680509" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45935 } }
I20260927 14:51:35.606731 2692 leader_election.cc:288] T 5cdbea9f22bc492cb0a10259e6b5319b P 7c70adccb37d4405976799ecc8680509 [CANDIDATE]: Term 1 election: Requested vote from peers 29daca91da71400b9bf7f441f06b045c (127.0.98.1:35643), be25711ef9c347959ab3b8f8ae136a07 (127.0.98.2:35577)
I20260927 14:51:35.606874 2575 raft_consensus.cc:2379] T 5cdbea9f22bc492cb0a10259e6b5319b P 7c70adccb37d4405976799ecc8680509 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 29daca91da71400b9bf7f441f06b045c in current term 1: Already voted for candidate 7c70adccb37d4405976799ecc8680509 in this term.
I20260927 14:51:35.607779 2227 leader_election.cc:302] T 5cdbea9f22bc492cb0a10259e6b5319b P 29daca91da71400b9bf7f441f06b045c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 29daca91da71400b9bf7f441f06b045c; no voters: 7c70adccb37d4405976799ecc8680509, be25711ef9c347959ab3b8f8ae136a07
I20260927 14:51:35.608069 2667 raft_consensus.cc:3037] T 5cdbea9f22bc492cb0a10259e6b5319b P 29daca91da71400b9bf7f441f06b045c [term 0 FOLLOWER]: Advancing to term 1
I20260927 14:51:35.608688 2643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:51:35.603510 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:42791 (local address 127.0.98.3:45935)
0927 14:51:35.603793 (+ 283us) server_negotiation.cc:206] Beginning negotiation
0927 14:51:35.603799 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 14:51:35.603814 (+ 15us) server_negotiation.cc:407] Connection header received
0927 14:51:35.603857 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:51:35.603861 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:51:35.603904 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:51:35.603987 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 14:51:35.604927 (+ 940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:51:35.605715 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:51:35.607414 (+ 1699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:51:35.608019 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:51:35.608089 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:51:35.608150 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0927 14:51:35.608499 (+ 349us) server_negotiation.cc:299] Negotiation successful
0927 14:51:35.608569 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":208}
I20260927 14:51:35.609014 2701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:51:35.602900 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:45935 (local address 127.0.98.2:42791)
0927 14:51:35.603247 (+ 347us) negotiation.cc:107] Waiting for socket to connect
0927 14:51:35.603261 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 14:51:35.603349 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:51:35.604658 (+ 1309us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:51:35.604662 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:51:35.604683 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 14:51:35.604798 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:51:35.604805 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:51:35.605898 (+ 1093us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:51:35.605902 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:51:35.607271 (+ 1369us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:51:35.607282 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:51:35.608255 (+ 973us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:51:35.608259 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:51:35.608362 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:51:35.608384 (+ 22us) client_negotiation.cc:769] Sending connection context
0927 14:51:35.608796 (+ 412us) client_negotiation.cc:240] Negotiation successful
0927 14:51:35.608873 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":275,"thread_start_us":111,"threads_started":1}
I20260927 14:51:35.609368 2667 raft_consensus.cc:2726] T 5cdbea9f22bc492cb0a10259e6b5319b P 29daca91da71400b9bf7f441f06b045c [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260927 14:51:35.609628 2434 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5cdbea9f22bc492cb0a10259e6b5319b" candidate_uuid: "7c70adccb37d4405976799ecc8680509" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "be25711ef9c347959ab3b8f8ae136a07"
I20260927 14:51:35.609740 2434 raft_consensus.cc:3037] T 5cdbea9f22bc492cb0a10259e6b5319b P be25711ef9c347959ab3b8f8ae136a07 [term 0 FOLLOWER]: Advancing to term 1
I20260927 14:51:35.610088 2575 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5cdbea9f22bc492cb0a10259e6b5319b" candidate_uuid: "be25711ef9c347959ab3b8f8ae136a07" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7c70adccb37d4405976799ecc8680509" is_pre_election: true
I20260927 14:51:35.610224 2575 raft_consensus.cc:2379] T 5cdbea9f22bc492cb0a10259e6b5319b P 7c70adccb37d4405976799ecc8680509 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate be25711ef9c347959ab3b8f8ae136a07 in current term 1: Already voted for candidate 7c70adccb37d4405976799ecc8680509 in this term.
I20260927 14:51:35.610554 2295 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5cdbea9f22bc492cb0a10259e6b5319b" candidate_uuid: "7c70adccb37d4405976799ecc8680509" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "29daca91da71400b9bf7f441f06b045c"
I20260927 14:51:35.611045 2434 raft_consensus.cc:2454] T 5cdbea9f22bc492cb0a10259e6b5319b P be25711ef9c347959ab3b8f8ae136a07 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7c70adccb37d4405976799ecc8680509 in term 1.
I20260927 14:51:35.611347 2508 leader_election.cc:302] T 5cdbea9f22bc492cb0a10259e6b5319b P 7c70adccb37d4405976799ecc8680509 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7c70adccb37d4405976799ecc8680509, be25711ef9c347959ab3b8f8ae136a07; no voters:
I20260927 14:51:35.611536 2692 raft_consensus.cc:2781] T 5cdbea9f22bc492cb0a10259e6b5319b P 7c70adccb37d4405976799ecc8680509 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 14:51:35.611644 2692 raft_consensus.cc:686] T 5cdbea9f22bc492cb0a10259e6b5319b P 7c70adccb37d4405976799ecc8680509 [term 1 LEADER]: Becoming Leader. State: Replica: 7c70adccb37d4405976799ecc8680509, State: Running, Role: LEADER
I20260927 14:51:35.611790 2295 raft_consensus.cc:2454] T 5cdbea9f22bc492cb0a10259e6b5319b P 29daca91da71400b9bf7f441f06b045c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7c70adccb37d4405976799ecc8680509 in term 1.
I20260927 14:51:35.611812 2692 consensus_queue.cc:237] T 5cdbea9f22bc492cb0a10259e6b5319b P 7c70adccb37d4405976799ecc8680509 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29daca91da71400b9bf7f441f06b045c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35643 } } peers { permanent_uuid: "be25711ef9c347959ab3b8f8ae136a07" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35577 } } peers { permanent_uuid: "7c70adccb37d4405976799ecc8680509" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45935 } }
I20260927 14:51:35.611932 2641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:51:35.604468 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:54393 (local address 127.0.98.1:35643)
0927 14:51:35.604666 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0927 14:51:35.604670 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 14:51:35.604684 (+ 14us) server_negotiation.cc:407] Connection header received
0927 14:51:35.604722 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:51:35.604725 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:51:35.604765 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:51:35.604832 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 14:51:35.606120 (+ 1288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:51:35.606865 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:51:35.610002 (+ 3137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:51:35.610651 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:51:35.610750 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:51:35.610810 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0927 14:51:35.611715 (+ 905us) server_negotiation.cc:299] Negotiation successful
0927 14:51:35.611796 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":119}
I20260927 14:51:35.612457 2700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:51:35.602493 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:35643 (local address 127.0.98.2:54393)
0927 14:51:35.602751 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0927 14:51:35.602768 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 14:51:35.602880 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:51:35.605831 (+ 2951us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:51:35.605835 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:51:35.605853 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 14:51:35.605977 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:51:35.605984 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:51:35.608329 (+ 2345us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:51:35.608334 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:51:35.609827 (+ 1493us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:51:35.609835 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:51:35.611342 (+ 1507us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:51:35.611348 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:51:35.611529 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:51:35.611557 (+ 28us) client_negotiation.cc:769] Sending connection context
0927 14:51:35.612224 (+ 667us) client_negotiation.cc:240] Negotiation successful
0927 14:51:35.612305 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":163,"thread_start_us":92,"threads_started":1}
I20260927 14:51:35.612926 2295 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5cdbea9f22bc492cb0a10259e6b5319b" candidate_uuid: "be25711ef9c347959ab3b8f8ae136a07" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "29daca91da71400b9bf7f441f06b045c" is_pre_election: true
I20260927 14:51:35.613102 2295 raft_consensus.cc:2379] T 5cdbea9f22bc492cb0a10259e6b5319b P 29daca91da71400b9bf7f441f06b045c [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate be25711ef9c347959ab3b8f8ae136a07 in current term 1: Already voted for candidate 7c70adccb37d4405976799ecc8680509 in this term.
I20260927 14:51:35.613425 2367 leader_election.cc:302] T 5cdbea9f22bc492cb0a10259e6b5319b P be25711ef9c347959ab3b8f8ae136a07 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: be25711ef9c347959ab3b8f8ae136a07; no voters: 29daca91da71400b9bf7f441f06b045c, 7c70adccb37d4405976799ecc8680509
I20260927 14:51:35.613626 2138 catalog_manager.cc:5495] T 5cdbea9f22bc492cb0a10259e6b5319b P 7c70adccb37d4405976799ecc8680509 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7c70adccb37d4405976799ecc8680509 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "7c70adccb37d4405976799ecc8680509" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29daca91da71400b9bf7f441f06b045c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35643 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "be25711ef9c347959ab3b8f8ae136a07" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35577 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7c70adccb37d4405976799ecc8680509" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45935 } health_report { overall_health: HEALTHY } } }
I20260927 14:51:35.613737 2650 raft_consensus.cc:2726] T 5cdbea9f22bc492cb0a10259e6b5319b P be25711ef9c347959ab3b8f8ae136a07 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260927 14:51:35.659713 2295 raft_consensus.cc:1261] T 5cdbea9f22bc492cb0a10259e6b5319b P 29daca91da71400b9bf7f441f06b045c [term 1 FOLLOWER]: Refusing update from remote peer 7c70adccb37d4405976799ecc8680509: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 14:51:35.659713 2434 raft_consensus.cc:1261] T 5cdbea9f22bc492cb0a10259e6b5319b P be25711ef9c347959ab3b8f8ae136a07 [term 1 FOLLOWER]: Refusing update from remote peer 7c70adccb37d4405976799ecc8680509: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 14:51:35.660153 2651 consensus_queue.cc:1035] T 5cdbea9f22bc492cb0a10259e6b5319b P 7c70adccb37d4405976799ecc8680509 [LEADER]: Connected to new peer: Peer: permanent_uuid: "29daca91da71400b9bf7f441f06b045c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35643 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 14:51:35.660321 2692 consensus_queue.cc:1035] T 5cdbea9f22bc492cb0a10259e6b5319b P 7c70adccb37d4405976799ecc8680509 [LEADER]: Connected to new peer: Peer: permanent_uuid: "be25711ef9c347959ab3b8f8ae136a07" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35577 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260927 14:51:35.673869 2673 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260927 14:51:37.222183 2136 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50588:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 14:51:37.222347 2136 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50588:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 14:51:37.224741 2136 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4939c7ba397e4ab48e64602dd2f1047c: Sending DeleteTablet for 3 replicas of tablet 5cdbea9f22bc492cb0a10259e6b5319b
I20260927 14:51:37.225339 2275 tablet_service.cc:1514] Processing DeleteTablet for tablet 5cdbea9f22bc492cb0a10259e6b5319b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 14:51:37 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:41838
I20260927 14:51:37.225364 2415 tablet_service.cc:1514] Processing DeleteTablet for tablet 5cdbea9f22bc492cb0a10259e6b5319b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 14:51:37 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:53790
I20260927 14:51:37.225397 2555 tablet_service.cc:1514] Processing DeleteTablet for tablet 5cdbea9f22bc492cb0a10259e6b5319b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 14:51:37 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:59184
I20260927 14:51:37.225683 2728 tablet_replica.cc:330] T 5cdbea9f22bc492cb0a10259e6b5319b P 7c70adccb37d4405976799ecc8680509: stopping tablet replica
I20260927 14:51:37.225809 2728 raft_consensus.cc:2229] T 5cdbea9f22bc492cb0a10259e6b5319b P 7c70adccb37d4405976799ecc8680509 [term 1 LEADER]: Raft consensus shutting down.
I20260927 14:51:37.226006 2728 raft_consensus.cc:2258] T 5cdbea9f22bc492cb0a10259e6b5319b P 7c70adccb37d4405976799ecc8680509 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 14:51:37.226389 2727 tablet_replica.cc:330] T 5cdbea9f22bc492cb0a10259e6b5319b P 29daca91da71400b9bf7f441f06b045c: stopping tablet replica
I20260927 14:51:37.226514 2727 raft_consensus.cc:2229] T 5cdbea9f22bc492cb0a10259e6b5319b P 29daca91da71400b9bf7f441f06b045c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 14:51:37.226585 2728 ts_tablet_manager.cc:1905] T 5cdbea9f22bc492cb0a10259e6b5319b P 7c70adccb37d4405976799ecc8680509: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 14:51:37.226670 2727 raft_consensus.cc:2258] T 5cdbea9f22bc492cb0a10259e6b5319b P 29daca91da71400b9bf7f441f06b045c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 14:51:37.226895 2729 tablet_replica.cc:330] T 5cdbea9f22bc492cb0a10259e6b5319b P be25711ef9c347959ab3b8f8ae136a07: stopping tablet replica
I20260927 14:51:37.227028 2729 raft_consensus.cc:2229] T 5cdbea9f22bc492cb0a10259e6b5319b P be25711ef9c347959ab3b8f8ae136a07 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 14:51:37.227141 2729 raft_consensus.cc:2258] T 5cdbea9f22bc492cb0a10259e6b5319b P be25711ef9c347959ab3b8f8ae136a07 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 14:51:37.227591 2727 ts_tablet_manager.cc:1905] T 5cdbea9f22bc492cb0a10259e6b5319b P 29daca91da71400b9bf7f441f06b045c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 14:51:37.229336 2728 ts_tablet_manager.cc:1918] T 5cdbea9f22bc492cb0a10259e6b5319b P 7c70adccb37d4405976799ecc8680509: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260927 14:51:37.229417 2728 log.cc:1196] T 5cdbea9f22bc492cb0a10259e6b5319b P 7c70adccb37d4405976799ecc8680509: Deleting WAL directory at /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790520684602251-392-0/minicluster-data/ts-2/wal/wals/5cdbea9f22bc492cb0a10259e6b5319b
I20260927 14:51:37.229645 2729 ts_tablet_manager.cc:1905] T 5cdbea9f22bc492cb0a10259e6b5319b P be25711ef9c347959ab3b8f8ae136a07: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 14:51:37.229710 2728 ts_tablet_manager.cc:1939] T 5cdbea9f22bc492cb0a10259e6b5319b P 7c70adccb37d4405976799ecc8680509: Deleting consensus metadata
I20260927 14:51:37.230373 2123 catalog_manager.cc:4841] TS 7c70adccb37d4405976799ecc8680509 (127.0.98.3:45935): tablet 5cdbea9f22bc492cb0a10259e6b5319b (table test-table [id=8ec7161675cd492b89c8baa5fc700a40]) successfully deleted
I20260927 14:51:37.230631 2727 ts_tablet_manager.cc:1918] T 5cdbea9f22bc492cb0a10259e6b5319b P 29daca91da71400b9bf7f441f06b045c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260927 14:51:37.230702 2727 log.cc:1196] T 5cdbea9f22bc492cb0a10259e6b5319b P 29daca91da71400b9bf7f441f06b045c: Deleting WAL directory at /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790520684602251-392-0/minicluster-data/ts-0/wal/wals/5cdbea9f22bc492cb0a10259e6b5319b
I20260927 14:51:37.230943 2727 ts_tablet_manager.cc:1939] T 5cdbea9f22bc492cb0a10259e6b5319b P 29daca91da71400b9bf7f441f06b045c: Deleting consensus metadata
I20260927 14:51:37.231567 2123 catalog_manager.cc:4841] TS 29daca91da71400b9bf7f441f06b045c (127.0.98.1:35643): tablet 5cdbea9f22bc492cb0a10259e6b5319b (table test-table [id=8ec7161675cd492b89c8baa5fc700a40]) successfully deleted
I20260927 14:51:37.231775 2729 ts_tablet_manager.cc:1918] T 5cdbea9f22bc492cb0a10259e6b5319b P be25711ef9c347959ab3b8f8ae136a07: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260927 14:51:37.231859 2729 log.cc:1196] T 5cdbea9f22bc492cb0a10259e6b5319b P be25711ef9c347959ab3b8f8ae136a07: Deleting WAL directory at /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790520684602251-392-0/minicluster-data/ts-1/wal/wals/5cdbea9f22bc492cb0a10259e6b5319b
I20260927 14:51:37.232097 2729 ts_tablet_manager.cc:1939] T 5cdbea9f22bc492cb0a10259e6b5319b P be25711ef9c347959ab3b8f8ae136a07: Deleting consensus metadata
I20260927 14:51:37.232664 2124 catalog_manager.cc:4841] TS be25711ef9c347959ab3b8f8ae136a07 (127.0.98.2:35577): tablet 5cdbea9f22bc492cb0a10259e6b5319b (table test-table [id=8ec7161675cd492b89c8baa5fc700a40]) successfully deleted
Sep 27 14:51:37 dist-test-slave-xk1g krb5kdc[2093](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790520697, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 27 14:51:37 dist-test-slave-xk1g krb5kdc[2093](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790520697, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 14:51:37.260560 2737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:51:37.245853 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50614 (local address 127.0.98.62:40817)
0927 14:51:37.246257 (+ 404us) server_negotiation.cc:206] Beginning negotiation
0927 14:51:37.246265 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 14:51:37.246279 (+ 14us) server_negotiation.cc:407] Connection header received
0927 14:51:37.246323 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:51:37.246325 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:51:37.246366 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:51:37.246452 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:51:37.247275 (+ 823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:51:37.247854 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:51:37.248754 (+ 900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:51:37.249011 (+ 257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:51:37.249212 (+ 201us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:51:37.257299 (+ 8087us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:51:37.257303 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:51:37.257305 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 14:51:37.257337 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:51:37.258902 (+ 1565us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:51:37.259332 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:51:37.259335 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:51:37.259336 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:51:37.259418 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:51:37.259725 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:51:37.259727 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:51:37.259728 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:51:37.259910 (+ 182us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:51:37.260027 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0927 14:51:37.260230 (+ 203us) server_negotiation.cc:299] Negotiation successful
0927 14:51:37.260336 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":320,"thread_start_us":98,"threads_started":1}
I20260927 14:51:37.263391 2136 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:50614:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 14:51:37.263922 2136 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 14:51:37.268510 2275 tablet_service.cc:1467] Processing CreateTablet for tablet 1981c318d1ad485d9a63e07f9a0b7c39 (DEFAULT_TABLE table=test-table [id=8e5ad3adf20f43ad8541cb15beeba443]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 14:51:37.268574 2415 tablet_service.cc:1467] Processing CreateTablet for tablet 1981c318d1ad485d9a63e07f9a0b7c39 (DEFAULT_TABLE table=test-table [id=8e5ad3adf20f43ad8541cb15beeba443]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 14:51:37.268736 2555 tablet_service.cc:1467] Processing CreateTablet for tablet 1981c318d1ad485d9a63e07f9a0b7c39 (DEFAULT_TABLE table=test-table [id=8e5ad3adf20f43ad8541cb15beeba443]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 14:51:37.268828 2275 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1981c318d1ad485d9a63e07f9a0b7c39. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 14:51:37.268899 2415 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1981c318d1ad485d9a63e07f9a0b7c39. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 14:51:37.268987 2555 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1981c318d1ad485d9a63e07f9a0b7c39. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 14:51:37.271054 2738 tablet_bootstrap.cc:492] T 1981c318d1ad485d9a63e07f9a0b7c39 P be25711ef9c347959ab3b8f8ae136a07: Bootstrap starting.
I20260927 14:51:37.271478 2740 tablet_bootstrap.cc:492] T 1981c318d1ad485d9a63e07f9a0b7c39 P 29daca91da71400b9bf7f441f06b045c: Bootstrap starting.
I20260927 14:51:37.271472 2739 tablet_bootstrap.cc:492] T 1981c318d1ad485d9a63e07f9a0b7c39 P 7c70adccb37d4405976799ecc8680509: Bootstrap starting.
I20260927 14:51:37.272310 2738 tablet_bootstrap.cc:654] T 1981c318d1ad485d9a63e07f9a0b7c39 P be25711ef9c347959ab3b8f8ae136a07: Neither blocks nor log segments found. Creating new log.
I20260927 14:51:37.272333 2739 tablet_bootstrap.cc:654] T 1981c318d1ad485d9a63e07f9a0b7c39 P 7c70adccb37d4405976799ecc8680509: Neither blocks nor log segments found. Creating new log.
I20260927 14:51:37.272333 2740 tablet_bootstrap.cc:654] T 1981c318d1ad485d9a63e07f9a0b7c39 P 29daca91da71400b9bf7f441f06b045c: Neither blocks nor log segments found. Creating new log.
I20260927 14:51:37.273559 2738 tablet_bootstrap.cc:492] T 1981c318d1ad485d9a63e07f9a0b7c39 P be25711ef9c347959ab3b8f8ae136a07: No bootstrap required, opened a new log
I20260927 14:51:37.273634 2738 ts_tablet_manager.cc:1397] T 1981c318d1ad485d9a63e07f9a0b7c39 P be25711ef9c347959ab3b8f8ae136a07: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260927 14:51:37.273824 2739 tablet_bootstrap.cc:492] T 1981c318d1ad485d9a63e07f9a0b7c39 P 7c70adccb37d4405976799ecc8680509: No bootstrap required, opened a new log
I20260927 14:51:37.273921 2739 ts_tablet_manager.cc:1397] T 1981c318d1ad485d9a63e07f9a0b7c39 P 7c70adccb37d4405976799ecc8680509: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 14:51:37.274152 2740 tablet_bootstrap.cc:492] T 1981c318d1ad485d9a63e07f9a0b7c39 P 29daca91da71400b9bf7f441f06b045c: No bootstrap required, opened a new log
I20260927 14:51:37.274147 2738 raft_consensus.cc:348] T 1981c318d1ad485d9a63e07f9a0b7c39 P be25711ef9c347959ab3b8f8ae136a07 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29daca91da71400b9bf7f441f06b045c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35643 } } peers { permanent_uuid: "be25711ef9c347959ab3b8f8ae136a07" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35577 } } peers { permanent_uuid: "7c70adccb37d4405976799ecc8680509" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45935 } }
I20260927 14:51:37.274228 2740 ts_tablet_manager.cc:1397] T 1981c318d1ad485d9a63e07f9a0b7c39 P 29daca91da71400b9bf7f441f06b045c: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260927 14:51:37.274264 2738 raft_consensus.cc:374] T 1981c318d1ad485d9a63e07f9a0b7c39 P be25711ef9c347959ab3b8f8ae136a07 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 14:51:37.274302 2738 raft_consensus.cc:729] T 1981c318d1ad485d9a63e07f9a0b7c39 P be25711ef9c347959ab3b8f8ae136a07 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: be25711ef9c347959ab3b8f8ae136a07, State: Initialized, Role: FOLLOWER
I20260927 14:51:37.274430 2738 consensus_queue.cc:260] T 1981c318d1ad485d9a63e07f9a0b7c39 P be25711ef9c347959ab3b8f8ae136a07 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29daca91da71400b9bf7f441f06b045c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35643 } } peers { permanent_uuid: "be25711ef9c347959ab3b8f8ae136a07" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35577 } } peers { permanent_uuid: "7c70adccb37d4405976799ecc8680509" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45935 } }
I20260927 14:51:37.274494 2739 raft_consensus.cc:348] T 1981c318d1ad485d9a63e07f9a0b7c39 P 7c70adccb37d4405976799ecc8680509 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29daca91da71400b9bf7f441f06b045c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35643 } } peers { permanent_uuid: "be25711ef9c347959ab3b8f8ae136a07" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35577 } } peers { permanent_uuid: "7c70adccb37d4405976799ecc8680509" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45935 } }
I20260927 14:51:37.274593 2739 raft_consensus.cc:374] T 1981c318d1ad485d9a63e07f9a0b7c39 P 7c70adccb37d4405976799ecc8680509 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 14:51:37.274631 2739 raft_consensus.cc:729] T 1981c318d1ad485d9a63e07f9a0b7c39 P 7c70adccb37d4405976799ecc8680509 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7c70adccb37d4405976799ecc8680509, State: Initialized, Role: FOLLOWER
I20260927 14:51:37.274607 2740 raft_consensus.cc:348] T 1981c318d1ad485d9a63e07f9a0b7c39 P 29daca91da71400b9bf7f441f06b045c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29daca91da71400b9bf7f441f06b045c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35643 } } peers { permanent_uuid: "be25711ef9c347959ab3b8f8ae136a07" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35577 } } peers { permanent_uuid: "7c70adccb37d4405976799ecc8680509" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45935 } }
I20260927 14:51:37.274708 2740 raft_consensus.cc:374] T 1981c318d1ad485d9a63e07f9a0b7c39 P 29daca91da71400b9bf7f441f06b045c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 14:51:37.274739 2740 raft_consensus.cc:729] T 1981c318d1ad485d9a63e07f9a0b7c39 P 29daca91da71400b9bf7f441f06b045c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 29daca91da71400b9bf7f441f06b045c, State: Initialized, Role: FOLLOWER
I20260927 14:51:37.274752 2739 consensus_queue.cc:260] T 1981c318d1ad485d9a63e07f9a0b7c39 P 7c70adccb37d4405976799ecc8680509 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29daca91da71400b9bf7f441f06b045c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35643 } } peers { permanent_uuid: "be25711ef9c347959ab3b8f8ae136a07" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35577 } } peers { permanent_uuid: "7c70adccb37d4405976799ecc8680509" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45935 } }
I20260927 14:51:37.274847 2738 ts_tablet_manager.cc:1428] T 1981c318d1ad485d9a63e07f9a0b7c39 P be25711ef9c347959ab3b8f8ae136a07: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260927 14:51:37.274883 2740 consensus_queue.cc:260] T 1981c318d1ad485d9a63e07f9a0b7c39 P 29daca91da71400b9bf7f441f06b045c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29daca91da71400b9bf7f441f06b045c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35643 } } peers { permanent_uuid: "be25711ef9c347959ab3b8f8ae136a07" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35577 } } peers { permanent_uuid: "7c70adccb37d4405976799ecc8680509" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45935 } }
I20260927 14:51:37.275023 2739 ts_tablet_manager.cc:1428] T 1981c318d1ad485d9a63e07f9a0b7c39 P 7c70adccb37d4405976799ecc8680509: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260927 14:51:37.275557 2740 ts_tablet_manager.cc:1428] T 1981c318d1ad485d9a63e07f9a0b7c39 P 29daca91da71400b9bf7f441f06b045c: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260927 14:51:37.387832 2744 raft_consensus.cc:482] T 1981c318d1ad485d9a63e07f9a0b7c39 P be25711ef9c347959ab3b8f8ae136a07 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 14:51:37.388011 2744 raft_consensus.cc:504] T 1981c318d1ad485d9a63e07f9a0b7c39 P be25711ef9c347959ab3b8f8ae136a07 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29daca91da71400b9bf7f441f06b045c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35643 } } peers { permanent_uuid: "be25711ef9c347959ab3b8f8ae136a07" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35577 } } peers { permanent_uuid: "7c70adccb37d4405976799ecc8680509" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45935 } }
I20260927 14:51:37.388805 2744 leader_election.cc:288] T 1981c318d1ad485d9a63e07f9a0b7c39 P be25711ef9c347959ab3b8f8ae136a07 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 29daca91da71400b9bf7f441f06b045c (127.0.98.1:35643), 7c70adccb37d4405976799ecc8680509 (127.0.98.3:45935)
I20260927 14:51:37.389034 2295 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1981c318d1ad485d9a63e07f9a0b7c39" candidate_uuid: "be25711ef9c347959ab3b8f8ae136a07" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "29daca91da71400b9bf7f441f06b045c" is_pre_election: true
I20260927 14:51:37.389233 2295 raft_consensus.cc:2454] T 1981c318d1ad485d9a63e07f9a0b7c39 P 29daca91da71400b9bf7f441f06b045c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate be25711ef9c347959ab3b8f8ae136a07 in term 0.
I20260927 14:51:37.389276 2575 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1981c318d1ad485d9a63e07f9a0b7c39" candidate_uuid: "be25711ef9c347959ab3b8f8ae136a07" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7c70adccb37d4405976799ecc8680509" is_pre_election: true
I20260927 14:51:37.389437 2575 raft_consensus.cc:2454] T 1981c318d1ad485d9a63e07f9a0b7c39 P 7c70adccb37d4405976799ecc8680509 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate be25711ef9c347959ab3b8f8ae136a07 in term 0.
I20260927 14:51:37.389541 2367 leader_election.cc:302] T 1981c318d1ad485d9a63e07f9a0b7c39 P be25711ef9c347959ab3b8f8ae136a07 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 29daca91da71400b9bf7f441f06b045c, be25711ef9c347959ab3b8f8ae136a07; no voters:
I20260927 14:51:37.389766 2744 raft_consensus.cc:2781] T 1981c318d1ad485d9a63e07f9a0b7c39 P be25711ef9c347959ab3b8f8ae136a07 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 14:51:37.389819 2744 raft_consensus.cc:482] T 1981c318d1ad485d9a63e07f9a0b7c39 P be25711ef9c347959ab3b8f8ae136a07 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 14:51:37.389859 2744 raft_consensus.cc:3037] T 1981c318d1ad485d9a63e07f9a0b7c39 P be25711ef9c347959ab3b8f8ae136a07 [term 0 FOLLOWER]: Advancing to term 1
I20260927 14:51:37.391062 2744 raft_consensus.cc:504] T 1981c318d1ad485d9a63e07f9a0b7c39 P be25711ef9c347959ab3b8f8ae136a07 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29daca91da71400b9bf7f441f06b045c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35643 } } peers { permanent_uuid: "be25711ef9c347959ab3b8f8ae136a07" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35577 } } peers { permanent_uuid: "7c70adccb37d4405976799ecc8680509" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45935 } }
I20260927 14:51:37.391384 2744 leader_election.cc:288] T 1981c318d1ad485d9a63e07f9a0b7c39 P be25711ef9c347959ab3b8f8ae136a07 [CANDIDATE]: Term 1 election: Requested vote from peers 29daca91da71400b9bf7f441f06b045c (127.0.98.1:35643), 7c70adccb37d4405976799ecc8680509 (127.0.98.3:45935)
I20260927 14:51:37.391770 2295 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1981c318d1ad485d9a63e07f9a0b7c39" candidate_uuid: "be25711ef9c347959ab3b8f8ae136a07" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "29daca91da71400b9bf7f441f06b045c"
I20260927 14:51:37.391834 2575 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1981c318d1ad485d9a63e07f9a0b7c39" candidate_uuid: "be25711ef9c347959ab3b8f8ae136a07" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7c70adccb37d4405976799ecc8680509"
I20260927 14:51:37.391892 2295 raft_consensus.cc:3037] T 1981c318d1ad485d9a63e07f9a0b7c39 P 29daca91da71400b9bf7f441f06b045c [term 0 FOLLOWER]: Advancing to term 1
I20260927 14:51:37.391942 2575 raft_consensus.cc:3037] T 1981c318d1ad485d9a63e07f9a0b7c39 P 7c70adccb37d4405976799ecc8680509 [term 0 FOLLOWER]: Advancing to term 1
I20260927 14:51:37.393020 2295 raft_consensus.cc:2454] T 1981c318d1ad485d9a63e07f9a0b7c39 P 29daca91da71400b9bf7f441f06b045c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate be25711ef9c347959ab3b8f8ae136a07 in term 1.
I20260927 14:51:37.393016 2575 raft_consensus.cc:2454] T 1981c318d1ad485d9a63e07f9a0b7c39 P 7c70adccb37d4405976799ecc8680509 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate be25711ef9c347959ab3b8f8ae136a07 in term 1.
I20260927 14:51:37.393308 2367 leader_election.cc:302] T 1981c318d1ad485d9a63e07f9a0b7c39 P be25711ef9c347959ab3b8f8ae136a07 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7c70adccb37d4405976799ecc8680509, be25711ef9c347959ab3b8f8ae136a07; no voters:
I20260927 14:51:37.393450 2744 raft_consensus.cc:2781] T 1981c318d1ad485d9a63e07f9a0b7c39 P be25711ef9c347959ab3b8f8ae136a07 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 14:51:37.393675 2744 raft_consensus.cc:686] T 1981c318d1ad485d9a63e07f9a0b7c39 P be25711ef9c347959ab3b8f8ae136a07 [term 1 LEADER]: Becoming Leader. State: Replica: be25711ef9c347959ab3b8f8ae136a07, State: Running, Role: LEADER
I20260927 14:51:37.393929 2744 consensus_queue.cc:237] T 1981c318d1ad485d9a63e07f9a0b7c39 P be25711ef9c347959ab3b8f8ae136a07 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29daca91da71400b9bf7f441f06b045c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35643 } } peers { permanent_uuid: "be25711ef9c347959ab3b8f8ae136a07" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35577 } } peers { permanent_uuid: "7c70adccb37d4405976799ecc8680509" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45935 } }
I20260927 14:51:37.396345 2137 catalog_manager.cc:5495] T 1981c318d1ad485d9a63e07f9a0b7c39 P be25711ef9c347959ab3b8f8ae136a07 reported cstate change: term changed from 0 to 1, leader changed from <none> to be25711ef9c347959ab3b8f8ae136a07 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "be25711ef9c347959ab3b8f8ae136a07" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29daca91da71400b9bf7f441f06b045c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35643 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "be25711ef9c347959ab3b8f8ae136a07" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35577 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7c70adccb37d4405976799ecc8680509" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45935 } health_report { overall_health: UNKNOWN } } }
I20260927 14:51:37.432219 2748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:51:37.428485 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42236 (local address 127.0.98.2:35577)
0927 14:51:37.428781 (+ 296us) server_negotiation.cc:206] Beginning negotiation
0927 14:51:37.428788 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 14:51:37.428798 (+ 10us) server_negotiation.cc:407] Connection header received
0927 14:51:37.428868 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:51:37.428870 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:51:37.428918 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:51:37.429053 (+ 135us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 14:51:37.429321 (+ 268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:51:37.429867 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:51:37.430735 (+ 868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:51:37.430981 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:51:37.431087 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:51:37.431363 (+ 276us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 14:51:37.431810 (+ 447us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 14:51:37.431936 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0927 14:51:37.431999 (+ 63us) server_negotiation.cc:299] Negotiation successful
0927 14:51:37.432052 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":224,"thread_start_us":124,"threads_started":1}
I20260927 14:51:37.442236 2752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:51:37.438862 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:44319 (local address 127.0.98.1:35643)
0927 14:51:37.439131 (+ 269us) server_negotiation.cc:206] Beginning negotiation
0927 14:51:37.439137 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 14:51:37.439152 (+ 15us) server_negotiation.cc:407] Connection header received
0927 14:51:37.439195 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:51:37.439197 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:51:37.439238 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:51:37.439316 (+ 78us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 14:51:37.439539 (+ 223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:51:37.440044 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:51:37.440965 (+ 921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:51:37.441217 (+ 252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:51:37.441310 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:51:37.441651 (+ 341us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 14:51:37.441725 (+ 74us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 14:51:37.441792 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0927 14:51:37.442067 (+ 275us) server_negotiation.cc:299] Negotiation successful
0927 14:51:37.442115 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":209,"thread_start_us":124,"threads_started":1}
I20260927 14:51:37.442252 2751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:51:37.438736 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:35643 (local address 127.0.98.2:44319)
0927 14:51:37.439014 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0927 14:51:37.439035 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0927 14:51:37.439112 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:51:37.439331 (+ 219us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:51:37.439333 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:51:37.439338 (+ 5us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 14:51:37.439435 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:51:37.439450 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:51:37.440173 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:51:37.440176 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:51:37.440826 (+ 650us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:51:37.440835 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:51:37.441352 (+ 517us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:51:37.441355 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:51:37.441458 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:51:37.441502 (+ 44us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 14:51:37.441808 (+ 306us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 14:51:37.441813 (+ 5us) client_negotiation.cc:769] Sending connection context
0927 14:51:37.442062 (+ 249us) client_negotiation.cc:240] Negotiation successful
0927 14:51:37.442134 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":102,"threads_started":1}
I20260927 14:51:37.448077 2575 raft_consensus.cc:1261] T 1981c318d1ad485d9a63e07f9a0b7c39 P 7c70adccb37d4405976799ecc8680509 [term 1 FOLLOWER]: Refusing update from remote peer be25711ef9c347959ab3b8f8ae136a07: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 14:51:37.448097 2295 raft_consensus.cc:1261] T 1981c318d1ad485d9a63e07f9a0b7c39 P 29daca91da71400b9bf7f441f06b045c [term 1 FOLLOWER]: Refusing update from remote peer be25711ef9c347959ab3b8f8ae136a07: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 14:51:37.448477 2745 consensus_queue.cc:1035] T 1981c318d1ad485d9a63e07f9a0b7c39 P be25711ef9c347959ab3b8f8ae136a07 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7c70adccb37d4405976799ecc8680509" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 14:51:37.448640 2744 consensus_queue.cc:1035] T 1981c318d1ad485d9a63e07f9a0b7c39 P be25711ef9c347959ab3b8f8ae136a07 [LEADER]: Connected to new peer: Peer: permanent_uuid: "29daca91da71400b9bf7f441f06b045c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35643 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 14:51:37.465793 2748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:51:37.462418 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:45133 (local address 127.0.98.2:35577)
0927 14:51:37.462540 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0927 14:51:37.462545 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 14:51:37.462720 (+ 175us) server_negotiation.cc:407] Connection header received
0927 14:51:37.462809 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:51:37.462811 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:51:37.462848 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:51:37.462955 (+ 107us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 14:51:37.463107 (+ 152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:51:37.463689 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:51:37.464764 (+ 1075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:51:37.465109 (+ 345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:51:37.465182 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:51:37.465453 (+ 271us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 14:51:37.465536 (+ 83us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 14:51:37.465596 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0927 14:51:37.465649 (+ 53us) server_negotiation.cc:299] Negotiation successful
0927 14:51:37.465692 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260927 14:51:37.465819 2757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:51:37.462377 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:35577 (local address 127.0.98.1:45133)
0927 14:51:37.462639 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0927 14:51:37.462658 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 14:51:37.462744 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:51:37.462955 (+ 211us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:51:37.462956 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:51:37.462961 (+ 5us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 14:51:37.463047 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:51:37.463051 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:51:37.463821 (+ 770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:51:37.463824 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:51:37.464615 (+ 791us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:51:37.464625 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:51:37.465271 (+ 646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:51:37.465274 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:51:37.465331 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:51:37.465352 (+ 21us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 14:51:37.465596 (+ 244us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 14:51:37.465599 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 14:51:37.465641 (+ 42us) client_negotiation.cc:240] Negotiation successful
0927 14:51:37.465699 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":117,"threads_started":1}
I20260927 14:51:37.487197 2135 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:50614:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 14:51:37.487380 2135 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:50614:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 14:51:37.489635 2135 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4939c7ba397e4ab48e64602dd2f1047c: Sending DeleteTablet for 3 replicas of tablet 1981c318d1ad485d9a63e07f9a0b7c39
I20260927 14:51:37.490393 2555 tablet_service.cc:1514] Processing DeleteTablet for tablet 1981c318d1ad485d9a63e07f9a0b7c39 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 14:51:37 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:59184
I20260927 14:51:37.490620 2415 tablet_service.cc:1514] Processing DeleteTablet for tablet 1981c318d1ad485d9a63e07f9a0b7c39 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 14:51:37 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:53790
I20260927 14:51:37.490665 2275 tablet_service.cc:1514] Processing DeleteTablet for tablet 1981c318d1ad485d9a63e07f9a0b7c39 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 14:51:37 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:41838
I20260927 14:51:37.490829 2728 tablet_replica.cc:330] T 1981c318d1ad485d9a63e07f9a0b7c39 P 7c70adccb37d4405976799ecc8680509: stopping tablet replica
I20260927 14:51:37.490830 2727 tablet_replica.cc:330] T 1981c318d1ad485d9a63e07f9a0b7c39 P 29daca91da71400b9bf7f441f06b045c: stopping tablet replica
I20260927 14:51:37.490964 2727 raft_consensus.cc:2229] T 1981c318d1ad485d9a63e07f9a0b7c39 P 29daca91da71400b9bf7f441f06b045c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 14:51:37.490967 2728 raft_consensus.cc:2229] T 1981c318d1ad485d9a63e07f9a0b7c39 P 7c70adccb37d4405976799ecc8680509 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 14:51:37.491094 2728 raft_consensus.cc:2258] T 1981c318d1ad485d9a63e07f9a0b7c39 P 7c70adccb37d4405976799ecc8680509 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 14:51:37.491094 2727 raft_consensus.cc:2258] T 1981c318d1ad485d9a63e07f9a0b7c39 P 29daca91da71400b9bf7f441f06b045c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 14:51:37.491271 2729 tablet_replica.cc:330] T 1981c318d1ad485d9a63e07f9a0b7c39 P be25711ef9c347959ab3b8f8ae136a07: stopping tablet replica
I20260927 14:51:37.491420 2729 raft_consensus.cc:2229] T 1981c318d1ad485d9a63e07f9a0b7c39 P be25711ef9c347959ab3b8f8ae136a07 [term 1 LEADER]: Raft consensus shutting down.
I20260927 14:51:37.491715 2729 raft_consensus.cc:2258] T 1981c318d1ad485d9a63e07f9a0b7c39 P be25711ef9c347959ab3b8f8ae136a07 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 14:51:37.491952 2727 ts_tablet_manager.cc:1905] T 1981c318d1ad485d9a63e07f9a0b7c39 P 29daca91da71400b9bf7f441f06b045c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 14:51:37.492473 2728 ts_tablet_manager.cc:1905] T 1981c318d1ad485d9a63e07f9a0b7c39 P 7c70adccb37d4405976799ecc8680509: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 14:51:37.492560 2729 ts_tablet_manager.cc:1905] T 1981c318d1ad485d9a63e07f9a0b7c39 P be25711ef9c347959ab3b8f8ae136a07: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 14:51:37.495227 2728 ts_tablet_manager.cc:1918] T 1981c318d1ad485d9a63e07f9a0b7c39 P 7c70adccb37d4405976799ecc8680509: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 14:51:37.495302 2728 log.cc:1196] T 1981c318d1ad485d9a63e07f9a0b7c39 P 7c70adccb37d4405976799ecc8680509: Deleting WAL directory at /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790520684602251-392-0/minicluster-data/ts-2/wal/wals/1981c318d1ad485d9a63e07f9a0b7c39
I20260927 14:51:37.495301 2727 ts_tablet_manager.cc:1918] T 1981c318d1ad485d9a63e07f9a0b7c39 P 29daca91da71400b9bf7f441f06b045c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 14:51:37.495373 2727 log.cc:1196] T 1981c318d1ad485d9a63e07f9a0b7c39 P 29daca91da71400b9bf7f441f06b045c: Deleting WAL directory at /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790520684602251-392-0/minicluster-data/ts-0/wal/wals/1981c318d1ad485d9a63e07f9a0b7c39
I20260927 14:51:37.495563 2728 ts_tablet_manager.cc:1939] T 1981c318d1ad485d9a63e07f9a0b7c39 P 7c70adccb37d4405976799ecc8680509: Deleting consensus metadata
I20260927 14:51:37.495766 2727 ts_tablet_manager.cc:1939] T 1981c318d1ad485d9a63e07f9a0b7c39 P 29daca91da71400b9bf7f441f06b045c: Deleting consensus metadata
I20260927 14:51:37.495766 2729 ts_tablet_manager.cc:1918] T 1981c318d1ad485d9a63e07f9a0b7c39 P be25711ef9c347959ab3b8f8ae136a07: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 14:51:37.495831 2729 log.cc:1196] T 1981c318d1ad485d9a63e07f9a0b7c39 P be25711ef9c347959ab3b8f8ae136a07: Deleting WAL directory at /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790520684602251-392-0/minicluster-data/ts-1/wal/wals/1981c318d1ad485d9a63e07f9a0b7c39
I20260927 14:51:37.496070 2729 ts_tablet_manager.cc:1939] T 1981c318d1ad485d9a63e07f9a0b7c39 P be25711ef9c347959ab3b8f8ae136a07: Deleting consensus metadata
I20260927 14:51:37.496407 2123 catalog_manager.cc:4841] TS 7c70adccb37d4405976799ecc8680509 (127.0.98.3:45935): tablet 1981c318d1ad485d9a63e07f9a0b7c39 (table test-table [id=8e5ad3adf20f43ad8541cb15beeba443]) successfully deleted
I20260927 14:51:37.496718 2124 catalog_manager.cc:4841] TS be25711ef9c347959ab3b8f8ae136a07 (127.0.98.2:35577): tablet 1981c318d1ad485d9a63e07f9a0b7c39 (table test-table [id=8e5ad3adf20f43ad8541cb15beeba443]) successfully deleted
I20260927 14:51:37.496982 2135 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:50614:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 14:51:37.497502 2135 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 14:51:37.498306 2123 catalog_manager.cc:4841] TS 29daca91da71400b9bf7f441f06b045c (127.0.98.1:35643): tablet 1981c318d1ad485d9a63e07f9a0b7c39 (table test-table [id=8e5ad3adf20f43ad8541cb15beeba443]) successfully deleted
I20260927 14:51:37.502223 2415 tablet_service.cc:1467] Processing CreateTablet for tablet a26b2127efe148439f9ae2794155531d (DEFAULT_TABLE table=test-table [id=37f18830bb234a609c41af9b297ecd30]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 14:51:37.502223 2275 tablet_service.cc:1467] Processing CreateTablet for tablet a26b2127efe148439f9ae2794155531d (DEFAULT_TABLE table=test-table [id=37f18830bb234a609c41af9b297ecd30]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 14:51:37.502496 2275 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a26b2127efe148439f9ae2794155531d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 14:51:37.502496 2415 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a26b2127efe148439f9ae2794155531d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 14:51:37.504734 2740 tablet_bootstrap.cc:492] T a26b2127efe148439f9ae2794155531d P 29daca91da71400b9bf7f441f06b045c: Bootstrap starting.
I20260927 14:51:37.504769 2738 tablet_bootstrap.cc:492] T a26b2127efe148439f9ae2794155531d P be25711ef9c347959ab3b8f8ae136a07: Bootstrap starting.
I20260927 14:51:37.504946 2555 tablet_service.cc:1467] Processing CreateTablet for tablet a26b2127efe148439f9ae2794155531d (DEFAULT_TABLE table=test-table [id=37f18830bb234a609c41af9b297ecd30]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 14:51:37.505374 2555 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a26b2127efe148439f9ae2794155531d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 14:51:37.505685 2740 tablet_bootstrap.cc:654] T a26b2127efe148439f9ae2794155531d P 29daca91da71400b9bf7f441f06b045c: Neither blocks nor log segments found. Creating new log.
I20260927 14:51:37.505749 2738 tablet_bootstrap.cc:654] T a26b2127efe148439f9ae2794155531d P be25711ef9c347959ab3b8f8ae136a07: Neither blocks nor log segments found. Creating new log.
I20260927 14:51:37.506927 2740 tablet_bootstrap.cc:492] T a26b2127efe148439f9ae2794155531d P 29daca91da71400b9bf7f441f06b045c: No bootstrap required, opened a new log
I20260927 14:51:37.507014 2740 ts_tablet_manager.cc:1397] T a26b2127efe148439f9ae2794155531d P 29daca91da71400b9bf7f441f06b045c: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 14:51:37.507412 2740 raft_consensus.cc:348] T a26b2127efe148439f9ae2794155531d P 29daca91da71400b9bf7f441f06b045c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29daca91da71400b9bf7f441f06b045c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35643 } } peers { permanent_uuid: "be25711ef9c347959ab3b8f8ae136a07" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35577 } } peers { permanent_uuid: "7c70adccb37d4405976799ecc8680509" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45935 } }
I20260927 14:51:37.507624 2740 raft_consensus.cc:374] T a26b2127efe148439f9ae2794155531d P 29daca91da71400b9bf7f441f06b045c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 14:51:37.507668 2740 raft_consensus.cc:729] T a26b2127efe148439f9ae2794155531d P 29daca91da71400b9bf7f441f06b045c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 29daca91da71400b9bf7f441f06b045c, State: Initialized, Role: FOLLOWER
I20260927 14:51:37.507803 2739 tablet_bootstrap.cc:492] T a26b2127efe148439f9ae2794155531d P 7c70adccb37d4405976799ecc8680509: Bootstrap starting.
I20260927 14:51:37.507793 2740 consensus_queue.cc:260] T a26b2127efe148439f9ae2794155531d P 29daca91da71400b9bf7f441f06b045c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29daca91da71400b9bf7f441f06b045c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35643 } } peers { permanent_uuid: "be25711ef9c347959ab3b8f8ae136a07" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35577 } } peers { permanent_uuid: "7c70adccb37d4405976799ecc8680509" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45935 } }
I20260927 14:51:37.508073 2740 ts_tablet_manager.cc:1428] T a26b2127efe148439f9ae2794155531d P 29daca91da71400b9bf7f441f06b045c: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260927 14:51:37.508551 2739 tablet_bootstrap.cc:654] T a26b2127efe148439f9ae2794155531d P 7c70adccb37d4405976799ecc8680509: Neither blocks nor log segments found. Creating new log.
I20260927 14:51:37.508745 2738 tablet_bootstrap.cc:492] T a26b2127efe148439f9ae2794155531d P be25711ef9c347959ab3b8f8ae136a07: No bootstrap required, opened a new log
I20260927 14:51:37.508802 2738 ts_tablet_manager.cc:1397] T a26b2127efe148439f9ae2794155531d P be25711ef9c347959ab3b8f8ae136a07: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260927 14:51:37.509393 2738 raft_consensus.cc:348] T a26b2127efe148439f9ae2794155531d P be25711ef9c347959ab3b8f8ae136a07 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29daca91da71400b9bf7f441f06b045c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35643 } } peers { permanent_uuid: "be25711ef9c347959ab3b8f8ae136a07" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35577 } } peers { permanent_uuid: "7c70adccb37d4405976799ecc8680509" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45935 } }
I20260927 14:51:37.509532 2738 raft_consensus.cc:374] T a26b2127efe148439f9ae2794155531d P be25711ef9c347959ab3b8f8ae136a07 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 14:51:37.509570 2738 raft_consensus.cc:729] T a26b2127efe148439f9ae2794155531d P be25711ef9c347959ab3b8f8ae136a07 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: be25711ef9c347959ab3b8f8ae136a07, State: Initialized, Role: FOLLOWER
I20260927 14:51:37.509687 2738 consensus_queue.cc:260] T a26b2127efe148439f9ae2794155531d P be25711ef9c347959ab3b8f8ae136a07 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29daca91da71400b9bf7f441f06b045c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35643 } } peers { permanent_uuid: "be25711ef9c347959ab3b8f8ae136a07" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35577 } } peers { permanent_uuid: "7c70adccb37d4405976799ecc8680509" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45935 } }
I20260927 14:51:37.509989 2739 tablet_bootstrap.cc:492] T a26b2127efe148439f9ae2794155531d P 7c70adccb37d4405976799ecc8680509: No bootstrap required, opened a new log
I20260927 14:51:37.509984 2738 ts_tablet_manager.cc:1428] T a26b2127efe148439f9ae2794155531d P be25711ef9c347959ab3b8f8ae136a07: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260927 14:51:37.510061 2739 ts_tablet_manager.cc:1397] T a26b2127efe148439f9ae2794155531d P 7c70adccb37d4405976799ecc8680509: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 14:51:37.510426 2739 raft_consensus.cc:348] T a26b2127efe148439f9ae2794155531d P 7c70adccb37d4405976799ecc8680509 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29daca91da71400b9bf7f441f06b045c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35643 } } peers { permanent_uuid: "be25711ef9c347959ab3b8f8ae136a07" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35577 } } peers { permanent_uuid: "7c70adccb37d4405976799ecc8680509" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45935 } }
I20260927 14:51:37.510519 2739 raft_consensus.cc:374] T a26b2127efe148439f9ae2794155531d P 7c70adccb37d4405976799ecc8680509 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 14:51:37.510547 2739 raft_consensus.cc:729] T a26b2127efe148439f9ae2794155531d P 7c70adccb37d4405976799ecc8680509 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7c70adccb37d4405976799ecc8680509, State: Initialized, Role: FOLLOWER
I20260927 14:51:37.510739 2739 consensus_queue.cc:260] T a26b2127efe148439f9ae2794155531d P 7c70adccb37d4405976799ecc8680509 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29daca91da71400b9bf7f441f06b045c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35643 } } peers { permanent_uuid: "be25711ef9c347959ab3b8f8ae136a07" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35577 } } peers { permanent_uuid: "7c70adccb37d4405976799ecc8680509" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45935 } }
I20260927 14:51:37.511024 2739 ts_tablet_manager.cc:1428] T a26b2127efe148439f9ae2794155531d P 7c70adccb37d4405976799ecc8680509: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260927 14:51:37.682603 2713 raft_consensus.cc:482] T a26b2127efe148439f9ae2794155531d P 29daca91da71400b9bf7f441f06b045c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 14:51:37.682747 2713 raft_consensus.cc:504] T a26b2127efe148439f9ae2794155531d P 29daca91da71400b9bf7f441f06b045c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29daca91da71400b9bf7f441f06b045c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35643 } } peers { permanent_uuid: "be25711ef9c347959ab3b8f8ae136a07" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35577 } } peers { permanent_uuid: "7c70adccb37d4405976799ecc8680509" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45935 } }
I20260927 14:51:37.683230 2713 leader_election.cc:288] T a26b2127efe148439f9ae2794155531d P 29daca91da71400b9bf7f441f06b045c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers be25711ef9c347959ab3b8f8ae136a07 (127.0.98.2:35577), 7c70adccb37d4405976799ecc8680509 (127.0.98.3:45935)
I20260927 14:51:37.683768 2575 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a26b2127efe148439f9ae2794155531d" candidate_uuid: "29daca91da71400b9bf7f441f06b045c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7c70adccb37d4405976799ecc8680509" is_pre_election: true
I20260927 14:51:37.683768 2434 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a26b2127efe148439f9ae2794155531d" candidate_uuid: "29daca91da71400b9bf7f441f06b045c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "be25711ef9c347959ab3b8f8ae136a07" is_pre_election: true
I20260927 14:51:37.683930 2575 raft_consensus.cc:2454] T a26b2127efe148439f9ae2794155531d P 7c70adccb37d4405976799ecc8680509 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 29daca91da71400b9bf7f441f06b045c in term 0.
I20260927 14:51:37.683930 2434 raft_consensus.cc:2454] T a26b2127efe148439f9ae2794155531d P be25711ef9c347959ab3b8f8ae136a07 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 29daca91da71400b9bf7f441f06b045c in term 0.
I20260927 14:51:37.684302 2228 leader_election.cc:302] T a26b2127efe148439f9ae2794155531d P 29daca91da71400b9bf7f441f06b045c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 29daca91da71400b9bf7f441f06b045c, be25711ef9c347959ab3b8f8ae136a07; no voters:
I20260927 14:51:37.684514 2713 raft_consensus.cc:2781] T a26b2127efe148439f9ae2794155531d P 29daca91da71400b9bf7f441f06b045c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 14:51:37.684577 2713 raft_consensus.cc:482] T a26b2127efe148439f9ae2794155531d P 29daca91da71400b9bf7f441f06b045c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 14:51:37.684624 2713 raft_consensus.cc:3037] T a26b2127efe148439f9ae2794155531d P 29daca91da71400b9bf7f441f06b045c [term 0 FOLLOWER]: Advancing to term 1
I20260927 14:51:37.685699 2713 raft_consensus.cc:504] T a26b2127efe148439f9ae2794155531d P 29daca91da71400b9bf7f441f06b045c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29daca91da71400b9bf7f441f06b045c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35643 } } peers { permanent_uuid: "be25711ef9c347959ab3b8f8ae136a07" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35577 } } peers { permanent_uuid: "7c70adccb37d4405976799ecc8680509" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45935 } }
I20260927 14:51:37.686040 2713 leader_election.cc:288] T a26b2127efe148439f9ae2794155531d P 29daca91da71400b9bf7f441f06b045c [CANDIDATE]: Term 1 election: Requested vote from peers be25711ef9c347959ab3b8f8ae136a07 (127.0.98.2:35577), 7c70adccb37d4405976799ecc8680509 (127.0.98.3:45935)
I20260927 14:51:37.686398 2434 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a26b2127efe148439f9ae2794155531d" candidate_uuid: "29daca91da71400b9bf7f441f06b045c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "be25711ef9c347959ab3b8f8ae136a07"
I20260927 14:51:37.686398 2575 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a26b2127efe148439f9ae2794155531d" candidate_uuid: "29daca91da71400b9bf7f441f06b045c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7c70adccb37d4405976799ecc8680509"
I20260927 14:51:37.686528 2575 raft_consensus.cc:3037] T a26b2127efe148439f9ae2794155531d P 7c70adccb37d4405976799ecc8680509 [term 0 FOLLOWER]: Advancing to term 1
I20260927 14:51:37.686528 2434 raft_consensus.cc:3037] T a26b2127efe148439f9ae2794155531d P be25711ef9c347959ab3b8f8ae136a07 [term 0 FOLLOWER]: Advancing to term 1
I20260927 14:51:37.687608 2434 raft_consensus.cc:2454] T a26b2127efe148439f9ae2794155531d P be25711ef9c347959ab3b8f8ae136a07 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 29daca91da71400b9bf7f441f06b045c in term 1.
I20260927 14:51:37.687608 2575 raft_consensus.cc:2454] T a26b2127efe148439f9ae2794155531d P 7c70adccb37d4405976799ecc8680509 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 29daca91da71400b9bf7f441f06b045c in term 1.
I20260927 14:51:37.687927 2228 leader_election.cc:302] T a26b2127efe148439f9ae2794155531d P 29daca91da71400b9bf7f441f06b045c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 29daca91da71400b9bf7f441f06b045c, 7c70adccb37d4405976799ecc8680509, be25711ef9c347959ab3b8f8ae136a07; no voters:
I20260927 14:51:37.688102 2713 raft_consensus.cc:2781] T a26b2127efe148439f9ae2794155531d P 29daca91da71400b9bf7f441f06b045c [term 1 FOLLOWER]: Leader election won for term 1
I20260927 14:51:37.688234 2713 raft_consensus.cc:686] T a26b2127efe148439f9ae2794155531d P 29daca91da71400b9bf7f441f06b045c [term 1 LEADER]: Becoming Leader. State: Replica: 29daca91da71400b9bf7f441f06b045c, State: Running, Role: LEADER
I20260927 14:51:37.688381 2713 consensus_queue.cc:237] T a26b2127efe148439f9ae2794155531d P 29daca91da71400b9bf7f441f06b045c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29daca91da71400b9bf7f441f06b045c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35643 } } peers { permanent_uuid: "be25711ef9c347959ab3b8f8ae136a07" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35577 } } peers { permanent_uuid: "7c70adccb37d4405976799ecc8680509" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45935 } }
I20260927 14:51:37.690060 2135 catalog_manager.cc:5495] T a26b2127efe148439f9ae2794155531d P 29daca91da71400b9bf7f441f06b045c reported cstate change: term changed from 0 to 1, leader changed from <none> to 29daca91da71400b9bf7f441f06b045c (127.0.98.1). New cstate: current_term: 1 leader_uuid: "29daca91da71400b9bf7f441f06b045c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29daca91da71400b9bf7f441f06b045c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35643 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "be25711ef9c347959ab3b8f8ae136a07" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35577 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7c70adccb37d4405976799ecc8680509" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45935 } health_report { overall_health: UNKNOWN } } }
I20260927 14:51:37.712018 2752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:51:37.709187 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44152 (local address 127.0.98.1:35643)
0927 14:51:37.709289 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0927 14:51:37.709293 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 14:51:37.709314 (+ 21us) server_negotiation.cc:407] Connection header received
0927 14:51:37.709356 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:51:37.709358 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:51:37.709398 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:51:37.709466 (+ 68us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 14:51:37.709654 (+ 188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:51:37.710151 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:51:37.711061 (+ 910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:51:37.711290 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:51:37.711364 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:51:37.711597 (+ 233us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 14:51:37.711659 (+ 62us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 14:51:37.711739 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0927 14:51:37.711864 (+ 125us) server_negotiation.cc:299] Negotiation successful
0927 14:51:37.711912 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260927 14:51:37.720379 2757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:51:37.717376 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:35643 (local address 127.0.98.1:42789)
0927 14:51:37.717502 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0927 14:51:37.717516 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 14:51:37.717616 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:51:37.717829 (+ 213us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:51:37.717831 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:51:37.717837 (+ 6us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 14:51:37.717993 (+ 156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:51:37.717999 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:51:37.718664 (+ 665us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:51:37.718670 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:51:37.719226 (+ 556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:51:37.719233 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:51:37.719640 (+ 407us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:51:37.719644 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:51:37.719771 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:51:37.719797 (+ 26us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 14:51:37.720066 (+ 269us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 14:51:37.720072 (+ 6us) client_negotiation.cc:769] Sending connection context
0927 14:51:37.720134 (+ 62us) client_negotiation.cc:240] Negotiation successful
0927 14:51:37.720245 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":44,"mutex_wait_us":102}
I20260927 14:51:37.720384 2752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:51:37.717541 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:42789 (local address 127.0.98.1:35643)
0927 14:51:37.717647 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0927 14:51:37.717654 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 14:51:37.717667 (+ 13us) server_negotiation.cc:407] Connection header received
0927 14:51:37.717705 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:51:37.717708 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:51:37.717748 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:51:37.717823 (+ 75us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 14:51:37.718089 (+ 266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:51:37.718564 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:51:37.719337 (+ 773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:51:37.719541 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:51:37.719643 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:51:37.719927 (+ 284us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 14:51:37.719983 (+ 56us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 14:51:37.720065 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0927 14:51:37.720149 (+ 84us) server_negotiation.cc:299] Negotiation successful
0927 14:51:37.720226 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":77,"server-negotiator.queue_time_us":47}
I20260927 14:51:37.726193 2575 raft_consensus.cc:1261] T a26b2127efe148439f9ae2794155531d P 7c70adccb37d4405976799ecc8680509 [term 1 FOLLOWER]: Refusing update from remote peer 29daca91da71400b9bf7f441f06b045c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 14:51:37.726411 2434 raft_consensus.cc:1261] T a26b2127efe148439f9ae2794155531d P be25711ef9c347959ab3b8f8ae136a07 [term 1 FOLLOWER]: Refusing update from remote peer 29daca91da71400b9bf7f441f06b045c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 14:51:37.726713 2713 consensus_queue.cc:1035] T a26b2127efe148439f9ae2794155531d P 29daca91da71400b9bf7f441f06b045c [LEADER]: Connected to new peer: Peer: permanent_uuid: "7c70adccb37d4405976799ecc8680509" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 14:51:37.726868 2667 consensus_queue.cc:1035] T a26b2127efe148439f9ae2794155531d P 29daca91da71400b9bf7f441f06b045c [LEADER]: Connected to new peer: Peer: permanent_uuid: "be25711ef9c347959ab3b8f8ae136a07" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35577 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260927 14:51:37.740695 2706 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260927 14:51:39.295678 2137 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:50614:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 14:51:39.295845 2137 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:50614:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 14:51:39.299512 2137 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4939c7ba397e4ab48e64602dd2f1047c: Sending DeleteTablet for 3 replicas of tablet a26b2127efe148439f9ae2794155531d
I20260927 14:51:39.300128 2275 tablet_service.cc:1514] Processing DeleteTablet for tablet a26b2127efe148439f9ae2794155531d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 14:51:39 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:41838
I20260927 14:51:39.300168 2415 tablet_service.cc:1514] Processing DeleteTablet for tablet a26b2127efe148439f9ae2794155531d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 14:51:39 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:53790
I20260927 14:51:39.300166 2555 tablet_service.cc:1514] Processing DeleteTablet for tablet a26b2127efe148439f9ae2794155531d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 14:51:39 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:59184
I20260927 14:51:39.300511 2789 tablet_replica.cc:330] T a26b2127efe148439f9ae2794155531d P be25711ef9c347959ab3b8f8ae136a07: stopping tablet replica
I20260927 14:51:39.300555 2790 tablet_replica.cc:330] T a26b2127efe148439f9ae2794155531d P 7c70adccb37d4405976799ecc8680509: stopping tablet replica
I20260927 14:51:39.300601 2788 tablet_replica.cc:330] T a26b2127efe148439f9ae2794155531d P 29daca91da71400b9bf7f441f06b045c: stopping tablet replica
I20260927 14:51:39.300652 2789 raft_consensus.cc:2229] T a26b2127efe148439f9ae2794155531d P be25711ef9c347959ab3b8f8ae136a07 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 14:51:39.300675 2790 raft_consensus.cc:2229] T a26b2127efe148439f9ae2794155531d P 7c70adccb37d4405976799ecc8680509 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 14:51:39.300701 2788 raft_consensus.cc:2229] T a26b2127efe148439f9ae2794155531d P 29daca91da71400b9bf7f441f06b045c [term 1 LEADER]: Raft consensus shutting down.
I20260927 14:51:39.300766 2789 raft_consensus.cc:2258] T a26b2127efe148439f9ae2794155531d P be25711ef9c347959ab3b8f8ae136a07 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 14:51:39.300788 2790 raft_consensus.cc:2258] T a26b2127efe148439f9ae2794155531d P 7c70adccb37d4405976799ecc8680509 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 14:51:39.300971 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6936m5/build/debug/bin/kudu with pid 2210
I20260927 14:51:39.302228 2790 ts_tablet_manager.cc:1905] T a26b2127efe148439f9ae2794155531d P 7c70adccb37d4405976799ecc8680509: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 14:51:39.305560 2789 ts_tablet_manager.cc:1905] T a26b2127efe148439f9ae2794155531d P be25711ef9c347959ab3b8f8ae136a07: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260927 14:51:39.308830 2123 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:35643 (error 108)
I20260927 14:51:39.308916 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6936m5/build/debug/bin/kudu with pid 2350
I20260927 14:51:39.310463 2792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:51:39.309980 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:35643 (local address 127.0.0.1:44156)
0927 14:51:39.310242 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0927 14:51:39.310334 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:35643: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":166,"thread_start_us":85,"threads_started":1}
W20260927 14:51:39.310590 2123 catalog_manager.cc:4584] TS 29daca91da71400b9bf7f441f06b045c (127.0.98.1:35643): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet a26b2127efe148439f9ae2794155531d: Network error: Client connection negotiation failed: client connection to 127.0.98.1:35643: connect: Connection refused (error 111)
I20260927 14:51:39.314965 2790 ts_tablet_manager.cc:1918] T a26b2127efe148439f9ae2794155531d P 7c70adccb37d4405976799ecc8680509: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260927 14:51:39.315070 2790 log.cc:1196] T a26b2127efe148439f9ae2794155531d P 7c70adccb37d4405976799ecc8680509: Deleting WAL directory at /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790520684602251-392-0/minicluster-data/ts-2/wal/wals/a26b2127efe148439f9ae2794155531d
I20260927 14:51:39.315366 2790 ts_tablet_manager.cc:1939] T a26b2127efe148439f9ae2794155531d P 7c70adccb37d4405976799ecc8680509: Deleting consensus metadata
I20260927 14:51:39.316200 2123 catalog_manager.cc:4841] TS 7c70adccb37d4405976799ecc8680509 (127.0.98.3:45935): tablet a26b2127efe148439f9ae2794155531d (table test-table [id=37f18830bb234a609c41af9b297ecd30]) successfully deleted
I20260927 14:51:39.320255 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6936m5/build/debug/bin/kudu with pid 2490
I20260927 14:51:39.322571 2792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:51:39.322295 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:35577 (local address 127.0.0.1:42246)
0927 14:51:39.322447 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0927 14:51:39.322503 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:35577: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
I20260927 14:51:39.326902 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6936m5/build/debug/bin/kudu with pid 2106
2026-09-27T14:51:39Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (6601 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790520684602251-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 14:51:39 dist-test-slave-xk1g krb5kdc[2795](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 14:51:39 dist-test-slave-xk1g krb5kdc[2795](info): set up 2 sockets
Sep 27 14:51:39 dist-test-slave-xk1g krb5kdc[2795](info): commencing operation
krb5kdc: starting...
W20260927 14:51:40.364959 392 mini_kdc.cc:121] Time spent starting KDC: real 1.015s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790520684602251-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790520684602251-392-0/krb5kdc/test-user.keytab.
Sep 27 14:51:40 dist-test-slave-xk1g krb5kdc[2795](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790520700, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T14:51:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T14:51:40Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790520684602251-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790520684602251-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 14:51:40.488525 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6936m5/build/debug/bin/kudu
/tmp/dist-test-task6936m5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790520684602251-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790520684602251-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790520684602251-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790520684602251-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40793
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790520684602251-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41977
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:40793
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790520684602251-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790520684602251-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790520684602251-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 14:51:40.595582 2808 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 14:51:40.595850 2808 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 14:51:40.595889 2808 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 14:51:40.599772 2808 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 14:51:40.599829 2808 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 14:51:40.599849 2808 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 14:51:40.599866 2808 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 14:51:40.599881 2808 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 14:51:40.605216 2808 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41977
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790520684602251-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790520684602251-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:40793
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790520684602251-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40793
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790520684602251-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2808
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790520684602251-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:15:17 UTC on ccycloud.quasar-ledxfe.root.comops.site
build id 4676134
I20260927 14:51:40.606178 2808 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 14:51:40.607093 2808 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 14:51:40.615444 2814 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 14:51:40.615461 2813 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 14:51:40.615512 2808 server_base.cc:1034] running on GCE node
W20260927 14:51:40.615451 2816 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 14:51:40.616253 2808 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 14:51:40.617350 2808 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 14:51:40.618536 2808 hybrid_clock.cc:648] HybridClock initialized: now 1790520700618516 us; error 32 us; skew 500 ppm
Sep 27 14:51:40 dist-test-slave-xk1g krb5kdc[2795](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790520700, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 14:51:40.621323 2808 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 14:51:40.622224 2808 webserver.cc:460] Webserver started at http://127.0.98.62:44555/ using document root <none> and password file <none>
I20260927 14:51:40.622785 2808 fs_manager.cc:359] Metadata directory not provided
I20260927 14:51:40.622871 2808 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 14:51:40.623095 2808 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 14:51:40.625149 2808 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790520684602251-392-0/minicluster-data/master-0/data/instance:
uuid: "e310050d3e704ff5aaa6bdea97cf7314"
format_stamp: "Formatted at 2026-09-27 14:51:40 on dist-test-slave-xk1g"
server_key: "8d76e27977e224f8130e2d97421773dd"
server_key_iv: "a3e921a4a34e7e60742ace9f7998093f"
server_key_version: "encryptionkey@0"
I20260927 14:51:40.625697 2808 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790520684602251-392-0/minicluster-data/master-0/wal/instance:
uuid: "e310050d3e704ff5aaa6bdea97cf7314"
format_stamp: "Formatted at 2026-09-27 14:51:40 on dist-test-slave-xk1g"
server_key: "8d76e27977e224f8130e2d97421773dd"
server_key_iv: "a3e921a4a34e7e60742ace9f7998093f"
server_key_version: "encryptionkey@0"
I20260927 14:51:40.629316 2808 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260927 14:51:40.631559 2823 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 14:51:40.632789 2808 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 14:51:40.632936 2808 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790520684602251-392-0/minicluster-data/master-0/data,/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790520684602251-392-0/minicluster-data/master-0/wal
uuid: "e310050d3e704ff5aaa6bdea97cf7314"
format_stamp: "Formatted at 2026-09-27 14:51:40 on dist-test-slave-xk1g"
server_key: "8d76e27977e224f8130e2d97421773dd"
server_key_iv: "a3e921a4a34e7e60742ace9f7998093f"
server_key_version: "encryptionkey@0"
I20260927 14:51:40.633046 2808 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790520684602251-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790520684602251-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790520684602251-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 14:51:40.644665 2808 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 14:51:40.647925 2808 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 14:51:40.648126 2808 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 14:51:40.656105 2808 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:40793
I20260927 14:51:40.656119 2875 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:40793 every 8 connection(s)
I20260927 14:51:40.657102 2808 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790520684602251-392-0/minicluster-data/master-0/data/info.pb
I20260927 14:51:40.659771 2876 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 14:51:40.664669 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6936m5/build/debug/bin/kudu as pid 2808
I20260927 14:51:40.664788 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790520684602251-392-0/minicluster-data/master-0/wal/instance
I20260927 14:51:40.665042 392 external_mini_cluster.cc:1428] Setting key a75cc8535dc80ed2392407bd683d59f7
I20260927 14:51:40.665467 2876 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e310050d3e704ff5aaa6bdea97cf7314: Bootstrap starting.
I20260927 14:51:40.667939 2876 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e310050d3e704ff5aaa6bdea97cf7314: Neither blocks nor log segments found. Creating new log.
I20260927 14:51:40.668653 2876 log.cc:824] T 00000000000000000000000000000000 P e310050d3e704ff5aaa6bdea97cf7314: Log is configured to *not* fsync() on all Append() calls
I20260927 14:51:40.670879 2876 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e310050d3e704ff5aaa6bdea97cf7314: No bootstrap required, opened a new log
Sep 27 14:51:40 dist-test-slave-xk1g krb5kdc[2795](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790520700, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 14:51:40.674777 2876 raft_consensus.cc:348] T 00000000000000000000000000000000 P e310050d3e704ff5aaa6bdea97cf7314 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e310050d3e704ff5aaa6bdea97cf7314" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40793 } }
I20260927 14:51:40.674988 2876 raft_consensus.cc:374] T 00000000000000000000000000000000 P e310050d3e704ff5aaa6bdea97cf7314 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 14:51:40.675050 2876 raft_consensus.cc:729] T 00000000000000000000000000000000 P e310050d3e704ff5aaa6bdea97cf7314 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e310050d3e704ff5aaa6bdea97cf7314, State: Initialized, Role: FOLLOWER
I20260927 14:51:40.675525 2876 consensus_queue.cc:260] T 00000000000000000000000000000000 P e310050d3e704ff5aaa6bdea97cf7314 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e310050d3e704ff5aaa6bdea97cf7314" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40793 } }
I20260927 14:51:40.675688 2876 raft_consensus.cc:388] T 00000000000000000000000000000000 P e310050d3e704ff5aaa6bdea97cf7314 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 14:51:40.675779 2876 raft_consensus.cc:482] T 00000000000000000000000000000000 P e310050d3e704ff5aaa6bdea97cf7314 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 14:51:40.675877 2876 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e310050d3e704ff5aaa6bdea97cf7314 [term 0 FOLLOWER]: Advancing to term 1
I20260927 14:51:40.677027 2876 raft_consensus.cc:504] T 00000000000000000000000000000000 P e310050d3e704ff5aaa6bdea97cf7314 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e310050d3e704ff5aaa6bdea97cf7314" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40793 } }
I20260927 14:51:40.677390 2876 leader_election.cc:302] T 00000000000000000000000000000000 P e310050d3e704ff5aaa6bdea97cf7314 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e310050d3e704ff5aaa6bdea97cf7314; no voters:
I20260927 14:51:40.677711 2876 leader_election.cc:288] T 00000000000000000000000000000000 P e310050d3e704ff5aaa6bdea97cf7314 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 14:51:40.677798 2881 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e310050d3e704ff5aaa6bdea97cf7314 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 14:51:40.678015 2881 raft_consensus.cc:686] T 00000000000000000000000000000000 P e310050d3e704ff5aaa6bdea97cf7314 [term 1 LEADER]: Becoming Leader. State: Replica: e310050d3e704ff5aaa6bdea97cf7314, State: Running, Role: LEADER
I20260927 14:51:40.678377 2881 consensus_queue.cc:237] T 00000000000000000000000000000000 P e310050d3e704ff5aaa6bdea97cf7314 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e310050d3e704ff5aaa6bdea97cf7314" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40793 } }
I20260927 14:51:40.678949 2876 sys_catalog.cc:564] T 00000000000000000000000000000000 P e310050d3e704ff5aaa6bdea97cf7314 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 14:51:40.680086 2883 sys_catalog.cc:455] T 00000000000000000000000000000000 P e310050d3e704ff5aaa6bdea97cf7314 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e310050d3e704ff5aaa6bdea97cf7314. Latest consensus state: current_term: 1 leader_uuid: "e310050d3e704ff5aaa6bdea97cf7314" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e310050d3e704ff5aaa6bdea97cf7314" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40793 } } }
I20260927 14:51:40.680513 2883 sys_catalog.cc:458] T 00000000000000000000000000000000 P e310050d3e704ff5aaa6bdea97cf7314 [sys.catalog]: This master's current role is: LEADER
I20260927 14:51:40.680384 2882 sys_catalog.cc:455] T 00000000000000000000000000000000 P e310050d3e704ff5aaa6bdea97cf7314 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e310050d3e704ff5aaa6bdea97cf7314" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e310050d3e704ff5aaa6bdea97cf7314" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40793 } } }
I20260927 14:51:40.680833 2882 sys_catalog.cc:458] T 00000000000000000000000000000000 P e310050d3e704ff5aaa6bdea97cf7314 [sys.catalog]: This master's current role is: LEADER
I20260927 14:51:40.681114 2879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:51:40.666565 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51996 (local address 127.0.98.62:40793)
0927 14:51:40.667055 (+ 490us) server_negotiation.cc:206] Beginning negotiation
0927 14:51:40.667071 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0927 14:51:40.667106 (+ 35us) server_negotiation.cc:407] Connection header received
0927 14:51:40.667724 (+ 618us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:51:40.667741 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:51:40.668120 (+ 379us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:51:40.668466 (+ 346us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:51:40.669307 (+ 841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:51:40.670754 (+ 1447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:51:40.671689 (+ 935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:51:40.672349 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:51:40.672516 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:51:40.675276 (+ 2760us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:51:40.675282 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:51:40.675297 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 14:51:40.675327 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:51:40.677577 (+ 2250us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:51:40.678138 (+ 561us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:51:40.678144 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:51:40.678150 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:51:40.678279 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:51:40.678572 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:51:40.678575 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:51:40.678577 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:51:40.679053 (+ 476us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:51:40.679240 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0927 14:51:40.679810 (+ 570us) server_negotiation.cc:299] Negotiation successful
0927 14:51:40.680105 (+ 295us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":346,"thread_start_us":147,"threads_started":1}
W20260927 14:51:40.684028 2896 catalog_manager.cc:1558] T 00000000000000000000000000000000 P e310050d3e704ff5aaa6bdea97cf7314: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 14:51:40.684113 2896 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 14:51:40.684185 2893 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 14:51:40.684855 2893 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 14:51:40.689654 2893 catalog_manager.cc:1347] Generated new cluster ID: 0d6f67e5a9b542d89386910a042aafeb
I20260927 14:51:40.689711 2893 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 14:51:40.698316 2893 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 14:51:40.699162 2893 catalog_manager.cc:1504] Loading token signing keys...
I20260927 14:51:40.703580 2893 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e310050d3e704ff5aaa6bdea97cf7314: Generated new TSK 0
I20260927 14:51:40.704217 2893 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790520684602251-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790520684602251-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260927 14:51:40.733923 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6936m5/build/debug/bin/kudu
/tmp/dist-test-task6936m5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790520684602251-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790520684602251-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790520684602251-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790520684602251-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40793
--keytab_file=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790520684602251-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41977
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790520684602251-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790520684602251-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790520684602251-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 14:51:40.842096 2902 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 14:51:40.842398 2902 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 14:51:40.842492 2902 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 14:51:40.846592 2902 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 14:51:40.846676 2902 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 14:51:40.846773 2902 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 14:51:40.852010 2902 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41977
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790520684602251-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790520684602251-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790520684602251-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790520684602251-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40793
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2902
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790520684602251-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:15:17 UTC on ccycloud.quasar-ledxfe.root.comops.site
build id 4676134
I20260927 14:51:40.853047 2902 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 14:51:40.854130 2902 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 14:51:40.863921 2902 server_base.cc:1034] running on GCE node
W20260927 14:51:40.863871 2907 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 14:51:40.863871 2908 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 14:51:40.863852 2910 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 14:51:40.864497 2902 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 14:51:40.865190 2902 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 14:51:40.866367 2902 hybrid_clock.cc:648] HybridClock initialized: now 1790520700866346 us; error 34 us; skew 500 ppm
Sep 27 14:51:40 dist-test-slave-xk1g krb5kdc[2795](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790520700, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 14:51:40.869170 2902 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 14:51:40.870054 2902 webserver.cc:460] Webserver started at http://127.0.98.1:38689/ using document root <none> and password file <none>
I20260927 14:51:40.870638 2902 fs_manager.cc:359] Metadata directory not provided
I20260927 14:51:40.870702 2902 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 14:51:40.870863 2902 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 14:51:40.872810 2902 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790520684602251-392-0/minicluster-data/ts-0/data/instance:
uuid: "5ea596cb958d4df5a68a604b4c282da9"
format_stamp: "Formatted at 2026-09-27 14:51:40 on dist-test-slave-xk1g"
server_key: "d7b55b88f115bcafc35241dbc3734727"
server_key_iv: "0292d06761ebb800f82ab08134de2f24"
server_key_version: "encryptionkey@0"
I20260927 14:51:40.873358 2902 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790520684602251-392-0/minicluster-data/ts-0/wal/instance:
uuid: "5ea596cb958d4df5a68a604b4c282da9"
format_stamp: "Formatted at 2026-09-27 14:51:40 on dist-test-slave-xk1g"
server_key: "d7b55b88f115bcafc35241dbc3734727"
server_key_iv: "0292d06761ebb800f82ab08134de2f24"
server_key_version: "encryptionkey@0"
I20260927 14:51:40.876881 2902 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260927 14:51:40.879391 2917 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 14:51:40.880659 2902 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 14:51:40.880797 2902 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790520684602251-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790520684602251-392-0/minicluster-data/ts-0/wal
uuid: "5ea596cb958d4df5a68a604b4c282da9"
format_stamp: "Formatted at 2026-09-27 14:51:40 on dist-test-slave-xk1g"
server_key: "d7b55b88f115bcafc35241dbc3734727"
server_key_iv: "0292d06761ebb800f82ab08134de2f24"
server_key_version: "encryptionkey@0"
I20260927 14:51:40.880918 2902 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790520684602251-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790520684602251-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790520684602251-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 14:51:40.894583 2902 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 14:51:40.897856 2902 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 14:51:40.898052 2902 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 14:51:40.898628 2902 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 14:51:40.899461 2902 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 14:51:40.899537 2902 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 14:51:40.899605 2902 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 14:51:40.899657 2902 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 14:51:40.908851 2902 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:34581
I20260927 14:51:40.908882 3030 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:34581 every 8 connection(s)
I20260927 14:51:40.909847 2902 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790520684602251-392-0/minicluster-data/ts-0/data/info.pb
Sep 27 14:51:40 dist-test-slave-xk1g krb5kdc[2795](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790520700, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 14:51:40.919955 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6936m5/build/debug/bin/kudu as pid 2902
I20260927 14:51:40.920101 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790520684602251-392-0/minicluster-data/ts-0/wal/instance
I20260927 14:51:40.920439 392 external_mini_cluster.cc:1428] Setting key fd9f71a2db3f9685e9786bf1e9596d0d
I20260927 14:51:40.923463 2879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:51:40.911631 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:37443 (local address 127.0.98.62:40793)
0927 14:51:40.911765 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0927 14:51:40.911769 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 14:51:40.912584 (+ 815us) server_negotiation.cc:407] Connection header received
0927 14:51:40.914026 (+ 1442us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:51:40.914030 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:51:40.914084 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:51:40.914195 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:51:40.915307 (+ 1112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:51:40.915833 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:51:40.916974 (+ 1141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:51:40.917280 (+ 306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:51:40.917393 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:51:40.919787 (+ 2394us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:51:40.919791 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:51:40.919793 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 14:51:40.919824 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:51:40.921457 (+ 1633us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:51:40.921946 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:51:40.921950 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:51:40.921952 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:51:40.922029 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:51:40.922374 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:51:40.922379 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:51:40.922380 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:51:40.922589 (+ 209us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:51:40.922698 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0927 14:51:40.923175 (+ 477us) server_negotiation.cc:299] Negotiation successful
0927 14:51:40.923294 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260927 14:51:40.924258 3033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:51:40.912007 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40793 (local address 127.0.98.1:37443)
0927 14:51:40.912443 (+ 436us) negotiation.cc:107] Waiting for socket to connect
0927 14:51:40.912483 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0927 14:51:40.913809 (+ 1326us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:51:40.914361 (+ 552us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:51:40.914367 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:51:40.914707 (+ 340us) client_negotiation.cc:189] Negotiated authn=SASL
0927 14:51:40.915158 (+ 451us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:51:40.915170 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:51:40.915976 (+ 806us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:51:40.915980 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:51:40.916843 (+ 863us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:51:40.916852 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:51:40.917437 (+ 585us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:51:40.917441 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:51:40.917606 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:51:40.918146 (+ 540us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 14:51:40.918161 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 14:51:40.919630 (+ 1469us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 14:51:40.921569 (+ 1939us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 14:51:40.921575 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 14:51:40.921589 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0927 14:51:40.921838 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 14:51:40.922121 (+ 283us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 14:51:40.922126 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 14:51:40.922128 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 14:51:40.922277 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 14:51:40.922697 (+ 420us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 14:51:40.922703 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 14:51:40.922941 (+ 238us) client_negotiation.cc:769] Sending connection context
0927 14:51:40.923125 (+ 184us) client_negotiation.cc:240] Negotiation successful
0927 14:51:40.923454 (+ 329us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":252,"thread_start_us":104,"threads_started":1}
I20260927 14:51:40.925580 3031 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40793
I20260927 14:51:40.925838 3031 heartbeater.cc:464] Registering TS with master...
I20260927 14:51:40.926425 3031 heartbeater.cc:511] Master 127.0.98.62:40793 requested a full tablet report, sending...
I20260927 14:51:40.928189 2835 ts_manager.cc:194] Registered new tserver with Master: 5ea596cb958d4df5a68a604b4c282da9 (127.0.98.1:34581)
I20260927 14:51:40.929845 2835 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:37443
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790520684602251-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790520684602251-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260927 14:51:40.948355 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6936m5/build/debug/bin/kudu
/tmp/dist-test-task6936m5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790520684602251-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790520684602251-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790520684602251-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790520684602251-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40793
--keytab_file=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790520684602251-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41977
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790520684602251-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790520684602251-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790520684602251-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 14:51:41.056631 3036 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 14:51:41.056921 3036 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 14:51:41.056965 3036 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 14:51:41.061305 3036 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 14:51:41.061363 3036 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 14:51:41.061457 3036 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 14:51:41.066748 3036 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41977
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790520684602251-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790520684602251-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790520684602251-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790520684602251-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40793
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3036
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790520684602251-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:15:17 UTC on ccycloud.quasar-ledxfe.root.comops.site
build id 4676134
I20260927 14:51:41.067838 3036 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 14:51:41.068840 3036 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 14:51:41.078590 3041 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 14:51:41.078590 3044 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 14:51:41.078815 3042 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 14:51:41.078675 3036 server_base.cc:1034] running on GCE node
I20260927 14:51:41.079254 3036 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 14:51:41.079962 3036 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 14:51:41.081139 3036 hybrid_clock.cc:648] HybridClock initialized: now 1790520701081124 us; error 29 us; skew 500 ppm
Sep 27 14:51:41 dist-test-slave-xk1g krb5kdc[2795](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790520701, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 14:51:41.083971 3036 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 14:51:41.084951 3036 webserver.cc:460] Webserver started at http://127.0.98.2:42569/ using document root <none> and password file <none>
I20260927 14:51:41.085569 3036 fs_manager.cc:359] Metadata directory not provided
I20260927 14:51:41.085631 3036 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 14:51:41.085841 3036 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 14:51:41.087927 3036 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790520684602251-392-0/minicluster-data/ts-1/data/instance:
uuid: "0213dc53e1e341d7a3e502aeda5b0e15"
format_stamp: "Formatted at 2026-09-27 14:51:41 on dist-test-slave-xk1g"
server_key: "5482015cb8cb735f8e4d1f79c4c5b4bd"
server_key_iv: "9e175836b3760a8e4aed6895c2a4b594"
server_key_version: "encryptionkey@0"
I20260927 14:51:41.088492 3036 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790520684602251-392-0/minicluster-data/ts-1/wal/instance:
uuid: "0213dc53e1e341d7a3e502aeda5b0e15"
format_stamp: "Formatted at 2026-09-27 14:51:41 on dist-test-slave-xk1g"
server_key: "5482015cb8cb735f8e4d1f79c4c5b4bd"
server_key_iv: "9e175836b3760a8e4aed6895c2a4b594"
server_key_version: "encryptionkey@0"
I20260927 14:51:41.092523 3036 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260927 14:51:41.094885 3051 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 14:51:41.096062 3036 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 14:51:41.096201 3036 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790520684602251-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790520684602251-392-0/minicluster-data/ts-1/wal
uuid: "0213dc53e1e341d7a3e502aeda5b0e15"
format_stamp: "Formatted at 2026-09-27 14:51:41 on dist-test-slave-xk1g"
server_key: "5482015cb8cb735f8e4d1f79c4c5b4bd"
server_key_iv: "9e175836b3760a8e4aed6895c2a4b594"
server_key_version: "encryptionkey@0"
I20260927 14:51:41.096352 3036 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790520684602251-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790520684602251-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790520684602251-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 14:51:41.109354 3036 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 14:51:41.112632 3036 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 14:51:41.112851 3036 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 14:51:41.113528 3036 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 14:51:41.114399 3036 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 14:51:41.114466 3036 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 14:51:41.114537 3036 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 14:51:41.114588 3036 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 14:51:41.123116 3036 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:42645
I20260927 14:51:41.123236 3164 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:42645 every 8 connection(s)
I20260927 14:51:41.124027 3036 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790520684602251-392-0/minicluster-data/ts-1/data/info.pb
I20260927 14:51:41.124531 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6936m5/build/debug/bin/kudu as pid 3036
I20260927 14:51:41.124656 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790520684602251-392-0/minicluster-data/ts-1/wal/instance
I20260927 14:51:41.124917 392 external_mini_cluster.cc:1428] Setting key 7ea82b7692e15975a4673553eeef9e97
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Sep 27 14:51:41 dist-test-slave-xk1g krb5kdc[2795](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790520701, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 14:51:41.141431 2879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:51:41.126039 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:49599 (local address 127.0.98.62:40793)
0927 14:51:41.126192 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0927 14:51:41.126197 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 14:51:41.127241 (+ 1044us) server_negotiation.cc:407] Connection header received
0927 14:51:41.128655 (+ 1414us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:51:41.128659 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:51:41.128721 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:51:41.128840 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:51:41.130186 (+ 1346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:51:41.130726 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:51:41.132005 (+ 1279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:51:41.132238 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:51:41.132365 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:51:41.137352 (+ 4987us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:51:41.137356 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:51:41.137358 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 14:51:41.137412 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:51:41.139216 (+ 1804us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:51:41.139783 (+ 567us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:51:41.139787 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:51:41.139789 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:51:41.139882 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:51:41.140234 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:51:41.140239 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:51:41.140241 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:51:41.140450 (+ 209us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:51:41.140553 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0927 14:51:41.141128 (+ 575us) server_negotiation.cc:299] Negotiation successful
0927 14:51:41.141252 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260927 14:51:41.142294 3167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:51:41.126579 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40793 (local address 127.0.98.2:49599)
0927 14:51:41.127067 (+ 488us) negotiation.cc:107] Waiting for socket to connect
0927 14:51:41.127121 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0927 14:51:41.128434 (+ 1313us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:51:41.129029 (+ 595us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:51:41.129036 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:51:41.129466 (+ 430us) client_negotiation.cc:189] Negotiated authn=SASL
0927 14:51:41.130036 (+ 570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:51:41.130054 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:51:41.130872 (+ 818us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:51:41.130876 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:51:41.131868 (+ 992us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:51:41.131879 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:51:41.132392 (+ 513us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:51:41.132397 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:51:41.132591 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:51:41.133297 (+ 706us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 14:51:41.133314 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 14:51:41.137161 (+ 3847us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 14:51:41.139366 (+ 2205us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 14:51:41.139373 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 14:51:41.139391 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0927 14:51:41.139668 (+ 277us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 14:51:41.139976 (+ 308us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 14:51:41.139980 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 14:51:41.139982 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 14:51:41.140136 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 14:51:41.140565 (+ 429us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 14:51:41.140574 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 14:51:41.140852 (+ 278us) client_negotiation.cc:769] Sending connection context
0927 14:51:41.141089 (+ 237us) client_negotiation.cc:240] Negotiation successful
0927 14:51:41.141444 (+ 355us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":285,"thread_start_us":138,"threads_started":1}
I20260927 14:51:41.143776 3165 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40793
I20260927 14:51:41.144075 3165 heartbeater.cc:464] Registering TS with master...
I20260927 14:51:41.144587 3165 heartbeater.cc:511] Master 127.0.98.62:40793 requested a full tablet report, sending...
I20260927 14:51:41.145714 2835 ts_manager.cc:194] Registered new tserver with Master: 0213dc53e1e341d7a3e502aeda5b0e15 (127.0.98.2:42645)
I20260927 14:51:41.146332 2835 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:49599
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790520684602251-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790520684602251-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260927 14:51:41.153309 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6936m5/build/debug/bin/kudu
/tmp/dist-test-task6936m5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790520684602251-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790520684602251-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790520684602251-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790520684602251-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40793
--keytab_file=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790520684602251-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41977
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790520684602251-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790520684602251-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790520684602251-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 14:51:41.263118 3170 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 14:51:41.263381 3170 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 14:51:41.263432 3170 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 14:51:41.267535 3170 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 14:51:41.267599 3170 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 14:51:41.267659 3170 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 14:51:41.273154 3170 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41977
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790520684602251-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790520684602251-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790520684602251-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790520684602251-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40793
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3170
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790520684602251-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:15:17 UTC on ccycloud.quasar-ledxfe.root.comops.site
build id 4676134
I20260927 14:51:41.274224 3170 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 14:51:41.275386 3170 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 14:51:41.285117 3178 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 14:51:41.285117 3176 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 14:51:41.285243 3175 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 14:51:41.285502 3170 server_base.cc:1034] running on GCE node
I20260927 14:51:41.285962 3170 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 14:51:41.286607 3170 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 14:51:41.287792 3170 hybrid_clock.cc:648] HybridClock initialized: now 1790520701287775 us; error 38 us; skew 500 ppm
Sep 27 14:51:41 dist-test-slave-xk1g krb5kdc[2795](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790520701, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 14:51:41.290705 3170 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 14:51:41.291581 3170 webserver.cc:460] Webserver started at http://127.0.98.3:36813/ using document root <none> and password file <none>
I20260927 14:51:41.292132 3170 fs_manager.cc:359] Metadata directory not provided
I20260927 14:51:41.292220 3170 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 14:51:41.292454 3170 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 14:51:41.294510 3170 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790520684602251-392-0/minicluster-data/ts-2/data/instance:
uuid: "d4d9de86efd44eedaae2ad6bdb83273b"
format_stamp: "Formatted at 2026-09-27 14:51:41 on dist-test-slave-xk1g"
server_key: "16837fed43a16b810b1fd3f63254fc5d"
server_key_iv: "fbdb4698065ed58411522d25ed81c9dd"
server_key_version: "encryptionkey@0"
I20260927 14:51:41.295071 3170 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790520684602251-392-0/minicluster-data/ts-2/wal/instance:
uuid: "d4d9de86efd44eedaae2ad6bdb83273b"
format_stamp: "Formatted at 2026-09-27 14:51:41 on dist-test-slave-xk1g"
server_key: "16837fed43a16b810b1fd3f63254fc5d"
server_key_iv: "fbdb4698065ed58411522d25ed81c9dd"
server_key_version: "encryptionkey@0"
I20260927 14:51:41.298702 3170 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260927 14:51:41.301191 3185 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 14:51:41.302361 3170 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 14:51:41.302501 3170 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790520684602251-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790520684602251-392-0/minicluster-data/ts-2/wal
uuid: "d4d9de86efd44eedaae2ad6bdb83273b"
format_stamp: "Formatted at 2026-09-27 14:51:41 on dist-test-slave-xk1g"
server_key: "16837fed43a16b810b1fd3f63254fc5d"
server_key_iv: "fbdb4698065ed58411522d25ed81c9dd"
server_key_version: "encryptionkey@0"
I20260927 14:51:41.302608 3170 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790520684602251-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790520684602251-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790520684602251-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 14:51:41.321734 3170 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 14:51:41.325095 3170 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 14:51:41.325325 3170 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 14:51:41.325946 3170 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 14:51:41.326778 3170 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 14:51:41.326851 3170 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 14:51:41.326920 3170 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 14:51:41.326953 3170 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 14:51:41.336009 3170 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:34615
I20260927 14:51:41.336117 3298 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:34615 every 8 connection(s)
I20260927 14:51:41.336947 3170 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790520684602251-392-0/minicluster-data/ts-2/data/info.pb
I20260927 14:51:41.339293 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6936m5/build/debug/bin/kudu as pid 3170
I20260927 14:51:41.339407 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790520684602251-392-0/minicluster-data/ts-2/wal/instance
I20260927 14:51:41.339740 392 external_mini_cluster.cc:1428] Setting key 3ca955c7698b41ab2135f9dc187ed677
Sep 27 14:51:41 dist-test-slave-xk1g krb5kdc[2795](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790520701, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 14:51:41.350898 2879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:51:41.338845 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:54929 (local address 127.0.98.62:40793)
0927 14:51:41.339036 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0927 14:51:41.339040 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 14:51:41.340008 (+ 968us) server_negotiation.cc:407] Connection header received
0927 14:51:41.341836 (+ 1828us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:51:41.341840 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:51:41.341897 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:51:41.341992 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:51:41.343106 (+ 1114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:51:41.343661 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:51:41.344797 (+ 1136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:51:41.345024 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:51:41.345191 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:51:41.347669 (+ 2478us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:51:41.347672 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:51:41.347674 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 14:51:41.347696 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:51:41.348981 (+ 1285us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:51:41.349527 (+ 546us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:51:41.349530 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:51:41.349531 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:51:41.349605 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:51:41.349932 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:51:41.349934 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:51:41.349935 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:51:41.350098 (+ 163us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:51:41.350171 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0927 14:51:41.350688 (+ 517us) server_negotiation.cc:299] Negotiation successful
0927 14:51:41.350783 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260927 14:51:41.351717 3301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:51:41.339308 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40793 (local address 127.0.98.3:54929)
0927 14:51:41.339833 (+ 525us) negotiation.cc:107] Waiting for socket to connect
0927 14:51:41.339896 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0927 14:51:41.341588 (+ 1692us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:51:41.342159 (+ 571us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:51:41.342166 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:51:41.342511 (+ 345us) client_negotiation.cc:189] Negotiated authn=SASL
0927 14:51:41.342966 (+ 455us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:51:41.342978 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:51:41.343780 (+ 802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:51:41.343783 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:51:41.344666 (+ 883us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:51:41.344675 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:51:41.345242 (+ 567us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:51:41.345246 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:51:41.345412 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:51:41.346051 (+ 639us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 14:51:41.346066 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 14:51:41.347457 (+ 1391us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 14:51:41.349139 (+ 1682us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 14:51:41.349144 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 14:51:41.349156 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0927 14:51:41.349434 (+ 278us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 14:51:41.349688 (+ 254us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 14:51:41.349690 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 14:51:41.349691 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 14:51:41.349833 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 14:51:41.350230 (+ 397us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 14:51:41.350237 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 14:51:41.350479 (+ 242us) client_negotiation.cc:769] Sending connection context
0927 14:51:41.350646 (+ 167us) client_negotiation.cc:240] Negotiation successful
0927 14:51:41.350916 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":298,"thread_start_us":141,"threads_started":1}
I20260927 14:51:41.353106 3299 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40793
I20260927 14:51:41.353410 3299 heartbeater.cc:464] Registering TS with master...
I20260927 14:51:41.353935 3299 heartbeater.cc:511] Master 127.0.98.62:40793 requested a full tablet report, sending...
I20260927 14:51:41.354868 2835 ts_manager.cc:194] Registered new tserver with Master: d4d9de86efd44eedaae2ad6bdb83273b (127.0.98.3:34615)
I20260927 14:51:41.355401 2835 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:54929
I20260927 14:51:41.365711 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260927 14:51:41.377961 2879 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 14:51:41.376482 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52002 (local address 127.0.98.62:40793)
0927 14:51:41.376623 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0927 14:51:41.376629 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 14:51:41.376743 (+ 114us) server_negotiation.cc:407] Connection header received
0927 14:51:41.376856 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:51:41.376858 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:51:41.376898 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:51:41.376986 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:51:41.377796 (+ 810us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52002: BlockingRecv error: recv got EOF from 127.0.0.1:52002 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
I20260927 14:51:41.379035 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6936m5/build/debug/bin/kudu with pid 2902
I20260927 14:51:41.384358 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6936m5/build/debug/bin/kudu with pid 3036
I20260927 14:51:41.389585 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6936m5/build/debug/bin/kudu with pid 3170
I20260927 14:51:41.394500 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6936m5/build/debug/bin/kudu with pid 2808
2026-09-27T14:51:41Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (2065 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790520684602251-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 14:51:41 dist-test-slave-xk1g krb5kdc[3311](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 14:51:41 dist-test-slave-xk1g krb5kdc[3311](info): set up 2 sockets
Sep 27 14:51:41 dist-test-slave-xk1g krb5kdc[3311](info): commencing operation
krb5kdc: starting...
W20260927 14:51:42.433145 392 mini_kdc.cc:121] Time spent starting KDC: real 1.017s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790520684602251-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790520684602251-392-0/krb5kdc/test-user.keytab.
Sep 27 14:51:42 dist-test-slave-xk1g krb5kdc[3311](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790520702, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T14:51:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T14:51:42Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790520684602251-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790520684602251-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 14:51:42.551853 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6936m5/build/debug/bin/kudu
/tmp/dist-test-task6936m5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790520684602251-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790520684602251-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790520684602251-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790520684602251-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44115
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790520684602251-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44249
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:44115
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790520684602251-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790520684602251-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790520684602251-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 14:51:42.663136 3324 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 14:51:42.663465 3324 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 14:51:42.663542 3324 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 14:51:42.667773 3324 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 14:51:42.667865 3324 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 14:51:42.667912 3324 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 14:51:42.667945 3324 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 14:51:42.667976 3324 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 14:51:42.673403 3324 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44249
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790520684602251-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790520684602251-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:44115
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790520684602251-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44115
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790520684602251-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3324
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790520684602251-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:15:17 UTC on ccycloud.quasar-ledxfe.root.comops.site
build id 4676134
I20260927 14:51:42.674587 3324 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 14:51:42.675576 3324 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 14:51:42.684118 3332 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 14:51:42.684216 3330 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 14:51:42.684137 3324 server_base.cc:1034] running on GCE node
W20260927 14:51:42.684161 3329 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 14:51:42.685042 3324 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 14:51:42.686278 3324 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 14:51:42.687501 3324 hybrid_clock.cc:648] HybridClock initialized: now 1790520702687474 us; error 47 us; skew 500 ppm
Sep 27 14:51:42 dist-test-slave-xk1g krb5kdc[3311](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790520702, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 14:51:42.693732 3324 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 14:51:42.694603 3324 webserver.cc:460] Webserver started at http://127.0.98.62:34939/ using document root <none> and password file <none>
I20260927 14:51:42.695142 3324 fs_manager.cc:359] Metadata directory not provided
I20260927 14:51:42.695201 3324 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 14:51:42.695363 3324 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 14:51:42.697431 3324 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790520684602251-392-0/minicluster-data/master-0/data/instance:
uuid: "e461cb4995584b828dc006408d668486"
format_stamp: "Formatted at 2026-09-27 14:51:42 on dist-test-slave-xk1g"
server_key: "2e4d08f3229eed7ca7403084d67b6ece"
server_key_iv: "a19565694e02efdd33000cba6bbe4415"
server_key_version: "encryptionkey@0"
I20260927 14:51:42.697921 3324 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790520684602251-392-0/minicluster-data/master-0/wal/instance:
uuid: "e461cb4995584b828dc006408d668486"
format_stamp: "Formatted at 2026-09-27 14:51:42 on dist-test-slave-xk1g"
server_key: "2e4d08f3229eed7ca7403084d67b6ece"
server_key_iv: "a19565694e02efdd33000cba6bbe4415"
server_key_version: "encryptionkey@0"
I20260927 14:51:42.701525 3324 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260927 14:51:42.703895 3339 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 14:51:42.705309 3324 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 14:51:42.705435 3324 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790520684602251-392-0/minicluster-data/master-0/data,/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790520684602251-392-0/minicluster-data/master-0/wal
uuid: "e461cb4995584b828dc006408d668486"
format_stamp: "Formatted at 2026-09-27 14:51:42 on dist-test-slave-xk1g"
server_key: "2e4d08f3229eed7ca7403084d67b6ece"
server_key_iv: "a19565694e02efdd33000cba6bbe4415"
server_key_version: "encryptionkey@0"
I20260927 14:51:42.705539 3324 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790520684602251-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790520684602251-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790520684602251-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 14:51:42.739282 3324 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 14:51:42.742347 3324 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 14:51:42.742554 3324 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 14:51:42.749677 3324 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:44115
I20260927 14:51:42.750056 3391 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:44115 every 8 connection(s)
I20260927 14:51:42.750671 3324 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790520684602251-392-0/minicluster-data/master-0/data/info.pb
I20260927 14:51:42.753536 3392 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 14:51:42.758271 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6936m5/build/debug/bin/kudu as pid 3324
I20260927 14:51:42.758430 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790520684602251-392-0/minicluster-data/master-0/wal/instance
I20260927 14:51:42.758687 392 external_mini_cluster.cc:1428] Setting key 046722d908b4c7568d6a1aaefc5144e4
I20260927 14:51:42.758940 3392 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e461cb4995584b828dc006408d668486: Bootstrap starting.
I20260927 14:51:42.761611 3392 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e461cb4995584b828dc006408d668486: Neither blocks nor log segments found. Creating new log.
I20260927 14:51:42.762486 3392 log.cc:824] T 00000000000000000000000000000000 P e461cb4995584b828dc006408d668486: Log is configured to *not* fsync() on all Append() calls
I20260927 14:51:42.764622 3392 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e461cb4995584b828dc006408d668486: No bootstrap required, opened a new log
Sep 27 14:51:42 dist-test-slave-xk1g krb5kdc[3311](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790520702, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 14:51:42.768587 3392 raft_consensus.cc:348] T 00000000000000000000000000000000 P e461cb4995584b828dc006408d668486 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e461cb4995584b828dc006408d668486" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44115 } }
I20260927 14:51:42.768829 3392 raft_consensus.cc:374] T 00000000000000000000000000000000 P e461cb4995584b828dc006408d668486 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 14:51:42.768910 3392 raft_consensus.cc:729] T 00000000000000000000000000000000 P e461cb4995584b828dc006408d668486 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e461cb4995584b828dc006408d668486, State: Initialized, Role: FOLLOWER
I20260927 14:51:42.769423 3392 consensus_queue.cc:260] T 00000000000000000000000000000000 P e461cb4995584b828dc006408d668486 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e461cb4995584b828dc006408d668486" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44115 } }
I20260927 14:51:42.769609 3392 raft_consensus.cc:388] T 00000000000000000000000000000000 P e461cb4995584b828dc006408d668486 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 14:51:42.769686 3392 raft_consensus.cc:482] T 00000000000000000000000000000000 P e461cb4995584b828dc006408d668486 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 14:51:42.769793 3392 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e461cb4995584b828dc006408d668486 [term 0 FOLLOWER]: Advancing to term 1
I20260927 14:51:42.770969 3392 raft_consensus.cc:504] T 00000000000000000000000000000000 P e461cb4995584b828dc006408d668486 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e461cb4995584b828dc006408d668486" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44115 } }
I20260927 14:51:42.771327 3392 leader_election.cc:302] T 00000000000000000000000000000000 P e461cb4995584b828dc006408d668486 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e461cb4995584b828dc006408d668486; no voters:
I20260927 14:51:42.771684 3392 leader_election.cc:288] T 00000000000000000000000000000000 P e461cb4995584b828dc006408d668486 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 14:51:42.771732 3397 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e461cb4995584b828dc006408d668486 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 14:51:42.772009 3397 raft_consensus.cc:686] T 00000000000000000000000000000000 P e461cb4995584b828dc006408d668486 [term 1 LEADER]: Becoming Leader. State: Replica: e461cb4995584b828dc006408d668486, State: Running, Role: LEADER
I20260927 14:51:42.772349 3397 consensus_queue.cc:237] T 00000000000000000000000000000000 P e461cb4995584b828dc006408d668486 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e461cb4995584b828dc006408d668486" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44115 } }
I20260927 14:51:42.773016 3392 sys_catalog.cc:564] T 00000000000000000000000000000000 P e461cb4995584b828dc006408d668486 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 14:51:42.774251 3399 sys_catalog.cc:455] T 00000000000000000000000000000000 P e461cb4995584b828dc006408d668486 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e461cb4995584b828dc006408d668486. Latest consensus state: current_term: 1 leader_uuid: "e461cb4995584b828dc006408d668486" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e461cb4995584b828dc006408d668486" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44115 } } }
I20260927 14:51:42.774341 3395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:51:42.760014 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54276 (local address 127.0.98.62:44115)
0927 14:51:42.760429 (+ 415us) server_negotiation.cc:206] Beginning negotiation
0927 14:51:42.760447 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0927 14:51:42.760488 (+ 41us) server_negotiation.cc:407] Connection header received
0927 14:51:42.761350 (+ 862us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:51:42.761373 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:51:42.761798 (+ 425us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:51:42.762183 (+ 385us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:51:42.763022 (+ 839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:51:42.764399 (+ 1377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:51:42.765380 (+ 981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:51:42.766055 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:51:42.766210 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:51:42.768942 (+ 2732us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:51:42.768950 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:51:42.768970 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 14:51:42.769002 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:51:42.771104 (+ 2102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:51:42.771560 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:51:42.771565 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:51:42.771569 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:51:42.771677 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:51:42.771999 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:51:42.772004 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:51:42.772006 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:51:42.772491 (+ 485us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:51:42.772678 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0927 14:51:42.773094 (+ 416us) server_negotiation.cc:299] Negotiation successful
0927 14:51:42.773413 (+ 319us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":273,"thread_start_us":110,"threads_started":1}
I20260927 14:51:42.774377 3399 sys_catalog.cc:458] T 00000000000000000000000000000000 P e461cb4995584b828dc006408d668486 [sys.catalog]: This master's current role is: LEADER
I20260927 14:51:42.774952 3398 sys_catalog.cc:455] T 00000000000000000000000000000000 P e461cb4995584b828dc006408d668486 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e461cb4995584b828dc006408d668486" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e461cb4995584b828dc006408d668486" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44115 } } }
I20260927 14:51:42.775049 3398 sys_catalog.cc:458] T 00000000000000000000000000000000 P e461cb4995584b828dc006408d668486 [sys.catalog]: This master's current role is: LEADER
I20260927 14:51:42.777384 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
W20260927 14:51:42.777812 3412 catalog_manager.cc:1558] T 00000000000000000000000000000000 P e461cb4995584b828dc006408d668486: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 14:51:42.777872 3412 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 14:51:42.777949 3406 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 14:51:42.778589 3406 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 14:51:42.784598 3406 catalog_manager.cc:1347] Generated new cluster ID: 12dc255bdddb4670b8f724ff6ea9e240
I20260927 14:51:42.784662 3406 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 14:51:42.795300 3406 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 14:51:42.796066 3395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:51:42.790387 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54286 (local address 127.0.98.62:44115)
0927 14:51:42.790493 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0927 14:51:42.790498 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 14:51:42.790672 (+ 174us) server_negotiation.cc:407] Connection header received
0927 14:51:42.790792 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:51:42.790796 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:51:42.790844 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:51:42.790932 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:51:42.791457 (+ 525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:51:42.792264 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:51:42.793415 (+ 1151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:51:42.793819 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:51:42.793930 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:51:42.795028 (+ 1098us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:51:42.795033 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:51:42.795038 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 14:51:42.795592 (+ 554us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:51:42.795627 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 14:51:42.795647 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:51:42.795736 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0927 14:51:42.795867 (+ 131us) server_negotiation.cc:299] Negotiation successful
0927 14:51:42.795925 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":40}
I20260927 14:51:42.796398 3406 catalog_manager.cc:1504] Loading token signing keys...
I20260927 14:51:42.801708 3406 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e461cb4995584b828dc006408d668486: Generated new TSK 0
I20260927 14:51:42.802467 3406 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 14:51:42.809975 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6936m5/build/debug/bin/kudu with pid 3324
2026-09-27T14:51:42Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1413 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790520684602251-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 14:51:42 dist-test-slave-xk1g krb5kdc[3425](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 14:51:42 dist-test-slave-xk1g krb5kdc[3425](info): set up 2 sockets
Sep 27 14:51:42 dist-test-slave-xk1g krb5kdc[3425](info): commencing operation
krb5kdc: starting...
W20260927 14:51:43.836994 392 mini_kdc.cc:121] Time spent starting KDC: real 1.007s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790520684602251-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790520684602251-392-0/krb5kdc/test-user.keytab.
Sep 27 14:51:43 dist-test-slave-xk1g krb5kdc[3425](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790520703, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T14:51:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T14:51:43Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790520684602251-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790520684602251-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 14:51:43.960505 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6936m5/build/debug/bin/kudu
/tmp/dist-test-task6936m5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790520684602251-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790520684602251-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790520684602251-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790520684602251-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43239
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790520684602251-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39249
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:43239
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790520684602251-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790520684602251-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790520684602251-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 14:51:44.068706 3438 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 14:51:44.069022 3438 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 14:51:44.069106 3438 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 14:51:44.073112 3438 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 14:51:44.073204 3438 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 14:51:44.073242 3438 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 14:51:44.073297 3438 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 14:51:44.073336 3438 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 14:51:44.078773 3438 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39249
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790520684602251-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790520684602251-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:43239
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790520684602251-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43239
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790520684602251-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3438
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790520684602251-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:15:17 UTC on ccycloud.quasar-ledxfe.root.comops.site
build id 4676134
I20260927 14:51:44.079823 3438 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 14:51:44.080821 3438 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 14:51:44.088975 3446 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 14:51:44.088994 3443 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 14:51:44.089125 3438 server_base.cc:1034] running on GCE node
W20260927 14:51:44.088925 3444 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 14:51:44.089779 3438 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 14:51:44.090823 3438 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 14:51:44.092016 3438 hybrid_clock.cc:648] HybridClock initialized: now 1790520704091988 us; error 49 us; skew 500 ppm
Sep 27 14:51:44 dist-test-slave-xk1g krb5kdc[3425](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790520704, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 14:51:44.094679 3438 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 14:51:44.095429 3438 webserver.cc:460] Webserver started at http://127.0.98.62:36863/ using document root <none> and password file <none>
I20260927 14:51:44.095932 3438 fs_manager.cc:359] Metadata directory not provided
I20260927 14:51:44.095986 3438 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 14:51:44.096136 3438 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 14:51:44.098133 3438 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790520684602251-392-0/minicluster-data/master-0/data/instance:
uuid: "166b07a51a364cb5a341bc8048a8e4f9"
format_stamp: "Formatted at 2026-09-27 14:51:44 on dist-test-slave-xk1g"
server_key: "52a78547ba85059eacac9e31f1b3b58b"
server_key_iv: "2ee9612cd93afec79e56f031c6f0b596"
server_key_version: "encryptionkey@0"
I20260927 14:51:44.098603 3438 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790520684602251-392-0/minicluster-data/master-0/wal/instance:
uuid: "166b07a51a364cb5a341bc8048a8e4f9"
format_stamp: "Formatted at 2026-09-27 14:51:44 on dist-test-slave-xk1g"
server_key: "52a78547ba85059eacac9e31f1b3b58b"
server_key_iv: "2ee9612cd93afec79e56f031c6f0b596"
server_key_version: "encryptionkey@0"
I20260927 14:51:44.102068 3438 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 14:51:44.104249 3453 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 14:51:44.105446 3438 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 14:51:44.105544 3438 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790520684602251-392-0/minicluster-data/master-0/data,/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790520684602251-392-0/minicluster-data/master-0/wal
uuid: "166b07a51a364cb5a341bc8048a8e4f9"
format_stamp: "Formatted at 2026-09-27 14:51:44 on dist-test-slave-xk1g"
server_key: "52a78547ba85059eacac9e31f1b3b58b"
server_key_iv: "2ee9612cd93afec79e56f031c6f0b596"
server_key_version: "encryptionkey@0"
I20260927 14:51:44.105621 3438 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790520684602251-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790520684602251-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790520684602251-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 14:51:44.135844 3438 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 14:51:44.138703 3438 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 14:51:44.138850 3438 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 14:51:44.145515 3438 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:43239
I20260927 14:51:44.145802 3505 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:43239 every 8 connection(s)
I20260927 14:51:44.146442 3438 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790520684602251-392-0/minicluster-data/master-0/data/info.pb
I20260927 14:51:44.149034 3506 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 14:51:44.154206 3506 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 166b07a51a364cb5a341bc8048a8e4f9: Bootstrap starting.
I20260927 14:51:44.156565 3506 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 166b07a51a364cb5a341bc8048a8e4f9: Neither blocks nor log segments found. Creating new log.
I20260927 14:51:44.156770 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6936m5/build/debug/bin/kudu as pid 3438
I20260927 14:51:44.156901 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790520684602251-392-0/minicluster-data/master-0/wal/instance
I20260927 14:51:44.157236 392 external_mini_cluster.cc:1428] Setting key 788daf6d90af2fb48686b41bdb999fa1
I20260927 14:51:44.157456 3506 log.cc:824] T 00000000000000000000000000000000 P 166b07a51a364cb5a341bc8048a8e4f9: Log is configured to *not* fsync() on all Append() calls
I20260927 14:51:44.159708 3506 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 166b07a51a364cb5a341bc8048a8e4f9: No bootstrap required, opened a new log
I20260927 14:51:44.163985 3506 raft_consensus.cc:348] T 00000000000000000000000000000000 P 166b07a51a364cb5a341bc8048a8e4f9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "166b07a51a364cb5a341bc8048a8e4f9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43239 } }
I20260927 14:51:44.164215 3506 raft_consensus.cc:374] T 00000000000000000000000000000000 P 166b07a51a364cb5a341bc8048a8e4f9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 14:51:44.164285 3506 raft_consensus.cc:729] T 00000000000000000000000000000000 P 166b07a51a364cb5a341bc8048a8e4f9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 166b07a51a364cb5a341bc8048a8e4f9, State: Initialized, Role: FOLLOWER
I20260927 14:51:44.164834 3506 consensus_queue.cc:260] T 00000000000000000000000000000000 P 166b07a51a364cb5a341bc8048a8e4f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "166b07a51a364cb5a341bc8048a8e4f9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43239 } }
I20260927 14:51:44.165036 3506 raft_consensus.cc:388] T 00000000000000000000000000000000 P 166b07a51a364cb5a341bc8048a8e4f9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 14:51:44.165154 3506 raft_consensus.cc:482] T 00000000000000000000000000000000 P 166b07a51a364cb5a341bc8048a8e4f9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 14:51:44.165263 3506 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 166b07a51a364cb5a341bc8048a8e4f9 [term 0 FOLLOWER]: Advancing to term 1
I20260927 14:51:44.166482 3506 raft_consensus.cc:504] T 00000000000000000000000000000000 P 166b07a51a364cb5a341bc8048a8e4f9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "166b07a51a364cb5a341bc8048a8e4f9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43239 } }
Sep 27 14:51:44 dist-test-slave-xk1g krb5kdc[3425](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790520703, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 14:51:44.166888 3506 leader_election.cc:302] T 00000000000000000000000000000000 P 166b07a51a364cb5a341bc8048a8e4f9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 166b07a51a364cb5a341bc8048a8e4f9; no voters:
I20260927 14:51:44.167239 3506 leader_election.cc:288] T 00000000000000000000000000000000 P 166b07a51a364cb5a341bc8048a8e4f9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 14:51:44.167304 3511 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 166b07a51a364cb5a341bc8048a8e4f9 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 14:51:44.167542 3511 raft_consensus.cc:686] T 00000000000000000000000000000000 P 166b07a51a364cb5a341bc8048a8e4f9 [term 1 LEADER]: Becoming Leader. State: Replica: 166b07a51a364cb5a341bc8048a8e4f9, State: Running, Role: LEADER
I20260927 14:51:44.167902 3511 consensus_queue.cc:237] T 00000000000000000000000000000000 P 166b07a51a364cb5a341bc8048a8e4f9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "166b07a51a364cb5a341bc8048a8e4f9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43239 } }
I20260927 14:51:44.168608 3506 sys_catalog.cc:564] T 00000000000000000000000000000000 P 166b07a51a364cb5a341bc8048a8e4f9 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 14:51:44.170535 3513 sys_catalog.cc:455] T 00000000000000000000000000000000 P 166b07a51a364cb5a341bc8048a8e4f9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 166b07a51a364cb5a341bc8048a8e4f9. Latest consensus state: current_term: 1 leader_uuid: "166b07a51a364cb5a341bc8048a8e4f9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "166b07a51a364cb5a341bc8048a8e4f9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43239 } } }
I20260927 14:51:44.170550 3512 sys_catalog.cc:455] T 00000000000000000000000000000000 P 166b07a51a364cb5a341bc8048a8e4f9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "166b07a51a364cb5a341bc8048a8e4f9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "166b07a51a364cb5a341bc8048a8e4f9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43239 } } }
I20260927 14:51:44.170675 3513 sys_catalog.cc:458] T 00000000000000000000000000000000 P 166b07a51a364cb5a341bc8048a8e4f9 [sys.catalog]: This master's current role is: LEADER
I20260927 14:51:44.170684 3512 sys_catalog.cc:458] T 00000000000000000000000000000000 P 166b07a51a364cb5a341bc8048a8e4f9 [sys.catalog]: This master's current role is: LEADER
I20260927 14:51:44.174010 3510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:51:44.158493 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41276 (local address 127.0.98.62:43239)
0927 14:51:44.158910 (+ 417us) server_negotiation.cc:206] Beginning negotiation
0927 14:51:44.158927 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0927 14:51:44.158964 (+ 37us) server_negotiation.cc:407] Connection header received
0927 14:51:44.159644 (+ 680us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:51:44.159662 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:51:44.160057 (+ 395us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:51:44.160365 (+ 308us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:51:44.161301 (+ 936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:51:44.162271 (+ 970us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:51:44.163639 (+ 1368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:51:44.164159 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:51:44.164422 (+ 263us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:51:44.167383 (+ 2961us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:51:44.167391 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:51:44.167409 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 14:51:44.167446 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:51:44.169705 (+ 2259us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:51:44.171031 (+ 1326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:51:44.171039 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:51:44.171046 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:51:44.171339 (+ 293us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:51:44.171711 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:51:44.171714 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:51:44.171716 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:51:44.172230 (+ 514us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:51:44.172457 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0927 14:51:44.172696 (+ 239us) server_negotiation.cc:299] Negotiation successful
0927 14:51:44.173009 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":269,"thread_start_us":113,"threads_started":1}
W20260927 14:51:44.174439 3525 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 166b07a51a364cb5a341bc8048a8e4f9: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 14:51:44.174518 3525 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 14:51:44.174582 3526 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 14:51:44.175428 3526 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 14:51:44.181043 3526 catalog_manager.cc:1347] Generated new cluster ID: b4ee9d0c9e02409683d310115ba99375
I20260927 14:51:44.181147 3526 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 14:51:44.193222 3526 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 14:51:44.194175 3526 catalog_manager.cc:1504] Loading token signing keys...
I20260927 14:51:44.203991 3526 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 166b07a51a364cb5a341bc8048a8e4f9: Generated new TSK 0
I20260927 14:51:44.204808 3526 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790520684602251-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790520684602251-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260927 14:51:44.233614 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6936m5/build/debug/bin/kudu
/tmp/dist-test-task6936m5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790520684602251-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790520684602251-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790520684602251-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790520684602251-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43239
--keytab_file=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790520684602251-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39249
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790520684602251-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790520684602251-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790520684602251-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 14:51:44.344739 3532 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 14:51:44.344991 3532 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 14:51:44.345037 3532 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 14:51:44.348953 3532 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 14:51:44.349018 3532 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 14:51:44.349160 3532 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 14:51:44.354295 3532 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39249
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790520684602251-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790520684602251-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790520684602251-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790520684602251-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43239
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3532
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790520684602251-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:15:17 UTC on ccycloud.quasar-ledxfe.root.comops.site
build id 4676134
I20260927 14:51:44.355365 3532 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 14:51:44.356371 3532 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 14:51:44.365861 3540 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 14:51:44.365890 3537 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 14:51:44.365891 3538 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 14:51:44.365943 3532 server_base.cc:1034] running on GCE node
I20260927 14:51:44.366518 3532 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 14:51:44.367116 3532 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 14:51:44.368325 3532 hybrid_clock.cc:648] HybridClock initialized: now 1790520704368302 us; error 28 us; skew 500 ppm
Sep 27 14:51:44 dist-test-slave-xk1g krb5kdc[3425](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790520704, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 14:51:44.371481 3532 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 14:51:44.372355 3532 webserver.cc:460] Webserver started at http://127.0.98.1:40775/ using document root <none> and password file <none>
I20260927 14:51:44.372895 3532 fs_manager.cc:359] Metadata directory not provided
I20260927 14:51:44.372983 3532 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 14:51:44.373229 3532 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 14:51:44.375411 3532 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790520684602251-392-0/minicluster-data/ts-0/data/instance:
uuid: "2c2268b6bd894a4a9b1f3c4fe1422697"
format_stamp: "Formatted at 2026-09-27 14:51:44 on dist-test-slave-xk1g"
server_key: "ccfd4772f2ff9e8cd5d05ec3d9d6fdd8"
server_key_iv: "a29e6c5e4584f624c842c7bbd5f129ca"
server_key_version: "encryptionkey@0"
I20260927 14:51:44.376104 3532 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790520684602251-392-0/minicluster-data/ts-0/wal/instance:
uuid: "2c2268b6bd894a4a9b1f3c4fe1422697"
format_stamp: "Formatted at 2026-09-27 14:51:44 on dist-test-slave-xk1g"
server_key: "ccfd4772f2ff9e8cd5d05ec3d9d6fdd8"
server_key_iv: "a29e6c5e4584f624c842c7bbd5f129ca"
server_key_version: "encryptionkey@0"
I20260927 14:51:44.380458 3532 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.001s
I20260927 14:51:44.383198 3547 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 14:51:44.384531 3532 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260927 14:51:44.384662 3532 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790520684602251-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790520684602251-392-0/minicluster-data/ts-0/wal
uuid: "2c2268b6bd894a4a9b1f3c4fe1422697"
format_stamp: "Formatted at 2026-09-27 14:51:44 on dist-test-slave-xk1g"
server_key: "ccfd4772f2ff9e8cd5d05ec3d9d6fdd8"
server_key_iv: "a29e6c5e4584f624c842c7bbd5f129ca"
server_key_version: "encryptionkey@0"
I20260927 14:51:44.384792 3532 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790520684602251-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790520684602251-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790520684602251-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 14:51:44.406199 3532 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 14:51:44.410326 3532 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 14:51:44.410591 3532 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 14:51:44.411253 3532 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 14:51:44.412212 3532 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 14:51:44.412286 3532 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 14:51:44.412376 3532 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 14:51:44.412433 3532 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 14:51:44.423696 3532 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:36897
I20260927 14:51:44.423764 3660 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:36897 every 8 connection(s)
I20260927 14:51:44.424758 3532 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790520684602251-392-0/minicluster-data/ts-0/data/info.pb
I20260927 14:51:44.429962 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6936m5/build/debug/bin/kudu as pid 3532
I20260927 14:51:44.430068 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790520684602251-392-0/minicluster-data/ts-0/wal/instance
I20260927 14:51:44.430347 392 external_mini_cluster.cc:1428] Setting key e6d76d58d8d5b4a6fffa74e9f3fcd7f2
Sep 27 14:51:44 dist-test-slave-xk1g krb5kdc[3425](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790520704, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260927 14:51:44.441031 3510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:51:44.426927 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:40719 (local address 127.0.98.62:43239)
0927 14:51:44.427099 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0927 14:51:44.427103 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 14:51:44.428009 (+ 906us) server_negotiation.cc:407] Connection header received
0927 14:51:44.429528 (+ 1519us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:51:44.429531 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:51:44.429587 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:51:44.429668 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:51:44.431102 (+ 1434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:51:44.431751 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:51:44.432927 (+ 1176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:51:44.433242 (+ 315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:51:44.433356 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:51:44.436743 (+ 3387us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:51:44.436751 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:51:44.436755 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 14:51:44.436793 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:51:44.438599 (+ 1806us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:51:44.439250 (+ 651us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:51:44.439257 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:51:44.439261 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:51:44.439359 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:51:44.439765 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:51:44.439768 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:51:44.439770 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:51:44.440010 (+ 240us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:51:44.440118 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0927 14:51:44.440777 (+ 659us) server_negotiation.cc:299] Negotiation successful
0927 14:51:44.440903 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260927 14:51:44.442207 3663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:51:44.427336 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43239 (local address 127.0.98.1:40719)
0927 14:51:44.427831 (+ 495us) negotiation.cc:107] Waiting for socket to connect
0927 14:51:44.427882 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0927 14:51:44.429305 (+ 1423us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:51:44.429839 (+ 534us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:51:44.429846 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:51:44.430298 (+ 452us) client_negotiation.cc:189] Negotiated authn=SASL
0927 14:51:44.430953 (+ 655us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:51:44.430970 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:51:44.431896 (+ 926us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:51:44.431900 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:51:44.432787 (+ 887us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:51:44.432796 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:51:44.433397 (+ 601us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:51:44.433401 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:51:44.433573 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:51:44.434213 (+ 640us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 14:51:44.434256 (+ 43us) client_negotiation.cc:656] calling sasl_client_start()
0927 14:51:44.436547 (+ 2291us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 14:51:44.438736 (+ 2189us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 14:51:44.438742 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 14:51:44.438757 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0927 14:51:44.439138 (+ 381us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 14:51:44.439483 (+ 345us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 14:51:44.439486 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 14:51:44.439488 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 14:51:44.439661 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 14:51:44.440154 (+ 493us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 14:51:44.440164 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 14:51:44.440522 (+ 358us) client_negotiation.cc:769] Sending connection context
0927 14:51:44.440700 (+ 178us) client_negotiation.cc:240] Negotiation successful
0927 14:51:44.441039 (+ 339us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":288,"thread_start_us":117,"threads_started":1}
I20260927 14:51:44.443562 3661 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43239
I20260927 14:51:44.443861 3661 heartbeater.cc:464] Registering TS with master...
I20260927 14:51:44.444406 3661 heartbeater.cc:511] Master 127.0.98.62:43239 requested a full tablet report, sending...
I20260927 14:51:44.446023 3470 ts_manager.cc:194] Registered new tserver with Master: 2c2268b6bd894a4a9b1f3c4fe1422697 (127.0.98.1:36897)
I20260927 14:51:44.447116 3470 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:40719
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790520684602251-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790520684602251-392-0/krb5kdc/kudu_127.0.98.2.keytab.
W20260927 14:51:44.683319 392 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.0.98.2: real 0.251s user 0.000s sys 0.003s
I20260927 14:51:44.683792 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6936m5/build/debug/bin/kudu
/tmp/dist-test-task6936m5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790520684602251-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790520684602251-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790520684602251-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790520684602251-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43239
--keytab_file=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790520684602251-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39249
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790520684602251-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790520684602251-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790520684602251-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 14:51:44.791785 3666 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 14:51:44.792088 3666 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 14:51:44.792165 3666 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 14:51:44.796197 3666 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 14:51:44.796295 3666 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 14:51:44.796394 3666 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 14:51:44.801613 3666 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39249
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790520684602251-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790520684602251-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790520684602251-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790520684602251-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43239
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3666
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790520684602251-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:15:17 UTC on ccycloud.quasar-ledxfe.root.comops.site
build id 4676134
I20260927 14:51:44.802692 3666 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 14:51:44.804029 3666 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 14:51:44.814363 3674 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 14:51:44.814438 3666 server_base.cc:1034] running on GCE node
W20260927 14:51:44.814353 3672 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 14:51:44.814352 3671 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 14:51:44.815213 3666 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 14:51:44.815801 3666 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 14:51:44.816989 3666 hybrid_clock.cc:648] HybridClock initialized: now 1790520704816971 us; error 35 us; skew 500 ppm
Sep 27 14:51:44 dist-test-slave-xk1g krb5kdc[3425](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790520704, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 14:51:44.819813 3666 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 14:51:44.820691 3666 webserver.cc:460] Webserver started at http://127.0.98.2:37735/ using document root <none> and password file <none>
I20260927 14:51:44.821324 3666 fs_manager.cc:359] Metadata directory not provided
I20260927 14:51:44.821385 3666 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 14:51:44.821594 3666 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 14:51:44.823559 3666 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790520684602251-392-0/minicluster-data/ts-1/data/instance:
uuid: "1abbe9d787d3462cbe31de0efe49bf30"
format_stamp: "Formatted at 2026-09-27 14:51:44 on dist-test-slave-xk1g"
server_key: "6cb0772b4a6e32acfddf140cec4cddd9"
server_key_iv: "234a20e4c6bf6bd942f76154cbf516bf"
server_key_version: "encryptionkey@0"
I20260927 14:51:44.824132 3666 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790520684602251-392-0/minicluster-data/ts-1/wal/instance:
uuid: "1abbe9d787d3462cbe31de0efe49bf30"
format_stamp: "Formatted at 2026-09-27 14:51:44 on dist-test-slave-xk1g"
server_key: "6cb0772b4a6e32acfddf140cec4cddd9"
server_key_iv: "234a20e4c6bf6bd942f76154cbf516bf"
server_key_version: "encryptionkey@0"
I20260927 14:51:44.827703 3666 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260927 14:51:44.830229 3681 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 14:51:44.831430 3666 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 14:51:44.831565 3666 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790520684602251-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790520684602251-392-0/minicluster-data/ts-1/wal
uuid: "1abbe9d787d3462cbe31de0efe49bf30"
format_stamp: "Formatted at 2026-09-27 14:51:44 on dist-test-slave-xk1g"
server_key: "6cb0772b4a6e32acfddf140cec4cddd9"
server_key_iv: "234a20e4c6bf6bd942f76154cbf516bf"
server_key_version: "encryptionkey@0"
I20260927 14:51:44.831674 3666 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790520684602251-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790520684602251-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790520684602251-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 14:51:44.843480 3666 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 14:51:44.846632 3666 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 14:51:44.846846 3666 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 14:51:44.847429 3666 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 14:51:44.848243 3666 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 14:51:44.848313 3666 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 14:51:44.848378 3666 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 14:51:44.848421 3666 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 14:51:44.857226 3666 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:33127
I20260927 14:51:44.857246 3794 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:33127 every 8 connection(s)
I20260927 14:51:44.858120 3666 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790520684602251-392-0/minicluster-data/ts-1/data/info.pb
I20260927 14:51:44.859925 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6936m5/build/debug/bin/kudu as pid 3666
I20260927 14:51:44.860059 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790520684602251-392-0/minicluster-data/ts-1/wal/instance
I20260927 14:51:44.860323 392 external_mini_cluster.cc:1428] Setting key 469a5d0160441886d7f53e26c666f7f3
Sep 27 14:51:44 dist-test-slave-xk1g krb5kdc[3425](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790520704, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260927 14:51:44.872547 3510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:51:44.859935 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:39549 (local address 127.0.98.62:43239)
0927 14:51:44.860088 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0927 14:51:44.860093 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 14:51:44.861011 (+ 918us) server_negotiation.cc:407] Connection header received
0927 14:51:44.862429 (+ 1418us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:51:44.862435 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:51:44.862495 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:51:44.862580 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:51:44.863690 (+ 1110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:51:44.864538 (+ 848us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:51:44.865727 (+ 1189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:51:44.866131 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:51:44.866260 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:51:44.868777 (+ 2517us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:51:44.868783 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:51:44.868788 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 14:51:44.868820 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:51:44.870446 (+ 1626us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:51:44.870957 (+ 511us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:51:44.870959 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:51:44.870961 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:51:44.871038 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:51:44.871400 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:51:44.871406 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:51:44.871410 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:51:44.871622 (+ 212us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:51:44.871700 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0927 14:51:44.872202 (+ 502us) server_negotiation.cc:299] Negotiation successful
0927 14:51:44.872348 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260927 14:51:44.873564 3797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:51:44.860360 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43239 (local address 127.0.98.2:39549)
0927 14:51:44.860854 (+ 494us) negotiation.cc:107] Waiting for socket to connect
0927 14:51:44.860905 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0927 14:51:44.862215 (+ 1310us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:51:44.862718 (+ 503us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:51:44.862725 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:51:44.863076 (+ 351us) client_negotiation.cc:189] Negotiated authn=SASL
0927 14:51:44.863532 (+ 456us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:51:44.863544 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:51:44.864666 (+ 1122us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:51:44.864668 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:51:44.865606 (+ 938us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:51:44.865615 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:51:44.866273 (+ 658us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:51:44.866276 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:51:44.866410 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:51:44.866988 (+ 578us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 14:51:44.867003 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 14:51:44.868609 (+ 1606us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 14:51:44.870592 (+ 1983us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 14:51:44.870597 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 14:51:44.870611 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0927 14:51:44.870874 (+ 263us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 14:51:44.871147 (+ 273us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 14:51:44.871150 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 14:51:44.871151 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 14:51:44.871288 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 14:51:44.871713 (+ 425us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 14:51:44.871718 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 14:51:44.871968 (+ 250us) client_negotiation.cc:769] Sending connection context
0927 14:51:44.872124 (+ 156us) client_negotiation.cc:240] Negotiation successful
0927 14:51:44.872439 (+ 315us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":291,"thread_start_us":139,"threads_started":1}
I20260927 14:51:44.874703 3795 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43239
I20260927 14:51:44.874960 3795 heartbeater.cc:464] Registering TS with master...
I20260927 14:51:44.875399 3795 heartbeater.cc:511] Master 127.0.98.62:43239 requested a full tablet report, sending...
I20260927 14:51:44.876499 3470 ts_manager.cc:194] Registered new tserver with Master: 1abbe9d787d3462cbe31de0efe49bf30 (127.0.98.2:33127)
I20260927 14:51:44.877415 3470 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:39549
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790520684602251-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790520684602251-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260927 14:51:44.888304 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6936m5/build/debug/bin/kudu
/tmp/dist-test-task6936m5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790520684602251-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790520684602251-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790520684602251-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790520684602251-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43239
--keytab_file=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790520684602251-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39249
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790520684602251-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790520684602251-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790520684602251-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 14:51:44.996618 3800 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 14:51:44.996871 3800 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 14:51:44.996932 3800 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 14:51:45.000844 3800 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 14:51:45.000928 3800 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 14:51:45.000988 3800 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 14:51:45.006112 3800 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39249
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790520684602251-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790520684602251-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790520684602251-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790520684602251-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43239
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3800
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790520684602251-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:15:17 UTC on ccycloud.quasar-ledxfe.root.comops.site
build id 4676134
I20260927 14:51:45.007042 3800 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 14:51:45.007961 3800 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 14:51:45.017233 3806 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 14:51:45.017278 3808 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 14:51:45.017309 3805 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 14:51:45.017418 3800 server_base.cc:1034] running on GCE node
I20260927 14:51:45.017815 3800 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 14:51:45.018427 3800 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 14:51:45.019609 3800 hybrid_clock.cc:648] HybridClock initialized: now 1790520705019581 us; error 41 us; skew 500 ppm
Sep 27 14:51:45 dist-test-slave-xk1g krb5kdc[3425](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790520705, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 14:51:45.022289 3800 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 14:51:45.023125 3800 webserver.cc:460] Webserver started at http://127.0.98.3:44593/ using document root <none> and password file <none>
I20260927 14:51:45.023664 3800 fs_manager.cc:359] Metadata directory not provided
I20260927 14:51:45.023751 3800 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 14:51:45.023985 3800 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 14:51:45.026012 3800 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790520684602251-392-0/minicluster-data/ts-2/data/instance:
uuid: "630f55e302554dd9a39d551a7624caaa"
format_stamp: "Formatted at 2026-09-27 14:51:45 on dist-test-slave-xk1g"
server_key: "0c7802acdedaea447dc96e54dab054a2"
server_key_iv: "b10ae86122bea67eac025b5a474bcca7"
server_key_version: "encryptionkey@0"
I20260927 14:51:45.026541 3800 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790520684602251-392-0/minicluster-data/ts-2/wal/instance:
uuid: "630f55e302554dd9a39d551a7624caaa"
format_stamp: "Formatted at 2026-09-27 14:51:45 on dist-test-slave-xk1g"
server_key: "0c7802acdedaea447dc96e54dab054a2"
server_key_iv: "b10ae86122bea67eac025b5a474bcca7"
server_key_version: "encryptionkey@0"
I20260927 14:51:45.030051 3800 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 14:51:45.032414 3815 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 14:51:45.033696 3800 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 14:51:45.033831 3800 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790520684602251-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790520684602251-392-0/minicluster-data/ts-2/wal
uuid: "630f55e302554dd9a39d551a7624caaa"
format_stamp: "Formatted at 2026-09-27 14:51:45 on dist-test-slave-xk1g"
server_key: "0c7802acdedaea447dc96e54dab054a2"
server_key_iv: "b10ae86122bea67eac025b5a474bcca7"
server_key_version: "encryptionkey@0"
I20260927 14:51:45.033958 3800 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790520684602251-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790520684602251-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790520684602251-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 14:51:45.046010 3800 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 14:51:45.048748 3800 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 14:51:45.048950 3800 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 14:51:45.049558 3800 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 14:51:45.050387 3800 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 14:51:45.050457 3800 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 14:51:45.050525 3800 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 14:51:45.050572 3800 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 14:51:45.059504 3800 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:38043
I20260927 14:51:45.059535 3928 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:38043 every 8 connection(s)
I20260927 14:51:45.060362 3800 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790520684602251-392-0/minicluster-data/ts-2/data/info.pb
I20260927 14:51:45.064388 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6936m5/build/debug/bin/kudu as pid 3800
I20260927 14:51:45.064510 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790520684602251-392-0/minicluster-data/ts-2/wal/instance
I20260927 14:51:45.064805 392 external_mini_cluster.cc:1428] Setting key 26522886f4f0c06e57e3447ef09a7e88
Sep 27 14:51:45 dist-test-slave-xk1g krb5kdc[3425](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790520705, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 14:51:45.073889 3510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:51:45.062153 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:37945 (local address 127.0.98.62:43239)
0927 14:51:45.062292 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0927 14:51:45.062295 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 14:51:45.063158 (+ 863us) server_negotiation.cc:407] Connection header received
0927 14:51:45.064722 (+ 1564us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:51:45.064725 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:51:45.064772 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:51:45.064847 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:51:45.066293 (+ 1446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:51:45.066818 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:51:45.067919 (+ 1101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:51:45.068144 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:51:45.068253 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:51:45.070722 (+ 2469us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:51:45.070725 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:51:45.070728 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 14:51:45.070753 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:51:45.071979 (+ 1226us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:51:45.072448 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:51:45.072454 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:51:45.072455 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:51:45.072535 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:51:45.072853 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:51:45.072858 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:51:45.072861 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:51:45.073038 (+ 177us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:51:45.073162 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0927 14:51:45.073621 (+ 459us) server_negotiation.cc:299] Negotiation successful
0927 14:51:45.073731 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260927 14:51:45.074649 3931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:51:45.062546 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43239 (local address 127.0.98.3:37945)
0927 14:51:45.063012 (+ 466us) negotiation.cc:107] Waiting for socket to connect
0927 14:51:45.063053 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0927 14:51:45.064495 (+ 1442us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:51:45.065028 (+ 533us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:51:45.065036 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:51:45.065536 (+ 500us) client_negotiation.cc:189] Negotiated authn=SASL
0927 14:51:45.066159 (+ 623us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:51:45.066180 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:51:45.066935 (+ 755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:51:45.066939 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:51:45.067792 (+ 853us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:51:45.067801 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:51:45.068286 (+ 485us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:51:45.068289 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:51:45.068452 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:51:45.069038 (+ 586us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 14:51:45.069097 (+ 59us) client_negotiation.cc:656] calling sasl_client_start()
0927 14:51:45.070568 (+ 1471us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 14:51:45.072088 (+ 1520us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 14:51:45.072093 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 14:51:45.072107 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0927 14:51:45.072347 (+ 240us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 14:51:45.072629 (+ 282us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 14:51:45.072632 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 14:51:45.072634 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 14:51:45.072762 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 14:51:45.073169 (+ 407us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 14:51:45.073177 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 14:51:45.073403 (+ 226us) client_negotiation.cc:769] Sending connection context
0927 14:51:45.073582 (+ 179us) client_negotiation.cc:240] Negotiation successful
0927 14:51:45.073889 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":255,"thread_start_us":108,"threads_started":1}
I20260927 14:51:45.075666 3929 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43239
I20260927 14:51:45.075870 3929 heartbeater.cc:464] Registering TS with master...
I20260927 14:51:45.076269 3929 heartbeater.cc:511] Master 127.0.98.62:43239 requested a full tablet report, sending...
I20260927 14:51:45.077250 3470 ts_manager.cc:194] Registered new tserver with Master: 630f55e302554dd9a39d551a7624caaa (127.0.98.3:38043)
I20260927 14:51:45.077986 3470 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:37945
I20260927 14:51:45.078446 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 27 14:51:45 dist-test-slave-xk1g krb5kdc[3425](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790520705, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Sep 27 14:51:45 dist-test-slave-xk1g krb5kdc[3425](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790520705, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 14:51:45.102772 3510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:51:45.094678 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41286 (local address 127.0.98.62:43239)
0927 14:51:45.094800 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0927 14:51:45.094803 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 14:51:45.094942 (+ 139us) server_negotiation.cc:407] Connection header received
0927 14:51:45.095033 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:51:45.095036 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:51:45.095074 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:51:45.095145 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:51:45.095768 (+ 623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:51:45.096270 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:51:45.097194 (+ 924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:51:45.097408 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:51:45.097500 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:51:45.099867 (+ 2367us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:51:45.099870 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:51:45.099872 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 14:51:45.099893 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:51:45.101143 (+ 1250us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:51:45.101541 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:51:45.101546 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:51:45.101548 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:51:45.101627 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:51:45.101898 (+ 271us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:51:45.101901 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:51:45.101903 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:51:45.102105 (+ 202us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:51:45.102256 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0927 14:51:45.102496 (+ 240us) server_negotiation.cc:299] Negotiation successful
0927 14:51:45.102608 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
W20260927 14:51:45.103130 3469 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:41286
I20260927 14:51:45.103932 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6936m5/build/debug/bin/kudu with pid 3532
I20260927 14:51:45.109251 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6936m5/build/debug/bin/kudu with pid 3666
I20260927 14:51:45.114523 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6936m5/build/debug/bin/kudu with pid 3800
I20260927 14:51:45.119654 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6936m5/build/debug/bin/kudu with pid 3438
2026-09-27T14:51:45Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (2311 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790520684602251-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 14:51:45 dist-test-slave-xk1g krb5kdc[3941](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 14:51:45 dist-test-slave-xk1g krb5kdc[3941](info): set up 2 sockets
Sep 27 14:51:45 dist-test-slave-xk1g krb5kdc[3941](info): commencing operation
krb5kdc: starting...
W20260927 14:51:46.157215 392 mini_kdc.cc:121] Time spent starting KDC: real 1.016s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790520684602251-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790520684602251-392-0/krb5kdc/test-user.keytab.
Sep 27 14:51:46 dist-test-slave-xk1g krb5kdc[3941](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790520706, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T14:51:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T14:51:46Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790520684602251-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790520684602251-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 14:51:46.281039 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6936m5/build/debug/bin/kudu
/tmp/dist-test-task6936m5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790520684602251-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790520684602251-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790520684602251-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790520684602251-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41789
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790520684602251-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40837
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:41789
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790520684602251-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790520684602251-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790520684602251-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 14:51:46.390389 3954 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 14:51:46.390650 3954 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 14:51:46.390688 3954 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 14:51:46.394703 3954 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 14:51:46.394771 3954 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 14:51:46.394790 3954 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 14:51:46.394807 3954 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 14:51:46.394822 3954 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 14:51:46.400192 3954 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40837
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790520684602251-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790520684602251-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:41789
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790520684602251-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41789
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790520684602251-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3954
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790520684602251-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:15:17 UTC on ccycloud.quasar-ledxfe.root.comops.site
build id 4676134
I20260927 14:51:46.401315 3954 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 14:51:46.402313 3954 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 14:51:46.410842 3959 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 14:51:46.410935 3954 server_base.cc:1034] running on GCE node
W20260927 14:51:46.410866 3962 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 14:51:46.410852 3960 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 14:51:46.411675 3954 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 14:51:46.412830 3954 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 14:51:46.414024 3954 hybrid_clock.cc:648] HybridClock initialized: now 1790520706414005 us; error 30 us; skew 500 ppm
Sep 27 14:51:46 dist-test-slave-xk1g krb5kdc[3941](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790520706, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 14:51:46.416795 3954 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 14:51:46.417776 3954 webserver.cc:460] Webserver started at http://127.0.98.62:35513/ using document root <none> and password file <none>
I20260927 14:51:46.418452 3954 fs_manager.cc:359] Metadata directory not provided
I20260927 14:51:46.418547 3954 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 14:51:46.418781 3954 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 14:51:46.420850 3954 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790520684602251-392-0/minicluster-data/master-0/data/instance:
uuid: "0ef21938a00a49ebac4117592a43cea7"
format_stamp: "Formatted at 2026-09-27 14:51:46 on dist-test-slave-xk1g"
server_key: "26d9c8baab3e97e0ceddb47317f3b5b8"
server_key_iv: "102b9ad3129f7be2f7b812c11415f33e"
server_key_version: "encryptionkey@0"
I20260927 14:51:46.421476 3954 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790520684602251-392-0/minicluster-data/master-0/wal/instance:
uuid: "0ef21938a00a49ebac4117592a43cea7"
format_stamp: "Formatted at 2026-09-27 14:51:46 on dist-test-slave-xk1g"
server_key: "26d9c8baab3e97e0ceddb47317f3b5b8"
server_key_iv: "102b9ad3129f7be2f7b812c11415f33e"
server_key_version: "encryptionkey@0"
I20260927 14:51:46.425145 3954 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260927 14:51:46.427663 3969 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 14:51:46.428953 3954 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260927 14:51:46.429129 3954 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790520684602251-392-0/minicluster-data/master-0/data,/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790520684602251-392-0/minicluster-data/master-0/wal
uuid: "0ef21938a00a49ebac4117592a43cea7"
format_stamp: "Formatted at 2026-09-27 14:51:46 on dist-test-slave-xk1g"
server_key: "26d9c8baab3e97e0ceddb47317f3b5b8"
server_key_iv: "102b9ad3129f7be2f7b812c11415f33e"
server_key_version: "encryptionkey@0"
I20260927 14:51:46.429255 3954 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790520684602251-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790520684602251-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790520684602251-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 14:51:46.451669 3954 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 14:51:46.454841 3954 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 14:51:46.455055 3954 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 14:51:46.462301 3954 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:41789
I20260927 14:51:46.462348 4021 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:41789 every 8 connection(s)
I20260927 14:51:46.463277 3954 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790520684602251-392-0/minicluster-data/master-0/data/info.pb
I20260927 14:51:46.466077 4022 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 14:51:46.467208 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6936m5/build/debug/bin/kudu as pid 3954
I20260927 14:51:46.467343 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790520684602251-392-0/minicluster-data/master-0/wal/instance
I20260927 14:51:46.467609 392 external_mini_cluster.cc:1428] Setting key 0cf3e2908114bdcae4f79e593dd99f92
I20260927 14:51:46.473320 4022 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0ef21938a00a49ebac4117592a43cea7: Bootstrap starting.
I20260927 14:51:46.476536 4022 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0ef21938a00a49ebac4117592a43cea7: Neither blocks nor log segments found. Creating new log.
I20260927 14:51:46.477623 4022 log.cc:824] T 00000000000000000000000000000000 P 0ef21938a00a49ebac4117592a43cea7: Log is configured to *not* fsync() on all Append() calls
Sep 27 14:51:46 dist-test-slave-xk1g krb5kdc[3941](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790520706, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 14:51:46.480319 4022 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0ef21938a00a49ebac4117592a43cea7: No bootstrap required, opened a new log
I20260927 14:51:46.484642 4025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:51:46.469155 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34572 (local address 127.0.98.62:41789)
0927 14:51:46.469577 (+ 422us) server_negotiation.cc:206] Beginning negotiation
0927 14:51:46.469597 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0927 14:51:46.469642 (+ 45us) server_negotiation.cc:407] Connection header received
0927 14:51:46.470507 (+ 865us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:51:46.470530 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:51:46.470961 (+ 431us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:51:46.471472 (+ 511us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:51:46.472180 (+ 708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:51:46.473670 (+ 1490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:51:46.474903 (+ 1233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:51:46.475644 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:51:46.475807 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:51:46.478809 (+ 3002us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:51:46.478816 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:51:46.478834 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 14:51:46.478889 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:51:46.481240 (+ 2351us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:51:46.481838 (+ 598us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:51:46.481847 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:51:46.481870 (+ 23us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:51:46.482036 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:51:46.482431 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:51:46.482437 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:51:46.482442 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:51:46.483038 (+ 596us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:51:46.483252 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0927 14:51:46.483628 (+ 376us) server_negotiation.cc:299] Negotiation successful
0927 14:51:46.483893 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":267,"thread_start_us":107,"threads_started":1}
I20260927 14:51:46.485369 4022 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0ef21938a00a49ebac4117592a43cea7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ef21938a00a49ebac4117592a43cea7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41789 } }
I20260927 14:51:46.485641 4022 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0ef21938a00a49ebac4117592a43cea7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 14:51:46.485725 4022 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0ef21938a00a49ebac4117592a43cea7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ef21938a00a49ebac4117592a43cea7, State: Initialized, Role: FOLLOWER
I20260927 14:51:46.486239 4022 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0ef21938a00a49ebac4117592a43cea7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ef21938a00a49ebac4117592a43cea7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41789 } }
I20260927 14:51:46.486404 4022 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0ef21938a00a49ebac4117592a43cea7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 14:51:46.486474 4022 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0ef21938a00a49ebac4117592a43cea7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 14:51:46.486570 4022 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0ef21938a00a49ebac4117592a43cea7 [term 0 FOLLOWER]: Advancing to term 1
I20260927 14:51:46.487686 4022 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0ef21938a00a49ebac4117592a43cea7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ef21938a00a49ebac4117592a43cea7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41789 } }
I20260927 14:51:46.487996 4022 leader_election.cc:302] T 00000000000000000000000000000000 P 0ef21938a00a49ebac4117592a43cea7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0ef21938a00a49ebac4117592a43cea7; no voters:
I20260927 14:51:46.488260 4022 leader_election.cc:288] T 00000000000000000000000000000000 P 0ef21938a00a49ebac4117592a43cea7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 14:51:46.488372 4027 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0ef21938a00a49ebac4117592a43cea7 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 14:51:46.488606 4027 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0ef21938a00a49ebac4117592a43cea7 [term 1 LEADER]: Becoming Leader. State: Replica: 0ef21938a00a49ebac4117592a43cea7, State: Running, Role: LEADER
I20260927 14:51:46.488956 4027 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0ef21938a00a49ebac4117592a43cea7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ef21938a00a49ebac4117592a43cea7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41789 } }
I20260927 14:51:46.489580 4022 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0ef21938a00a49ebac4117592a43cea7 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 14:51:46.490657 4029 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0ef21938a00a49ebac4117592a43cea7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0ef21938a00a49ebac4117592a43cea7. Latest consensus state: current_term: 1 leader_uuid: "0ef21938a00a49ebac4117592a43cea7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ef21938a00a49ebac4117592a43cea7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41789 } } }
I20260927 14:51:46.490803 4029 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0ef21938a00a49ebac4117592a43cea7 [sys.catalog]: This master's current role is: LEADER
I20260927 14:51:46.491106 4028 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0ef21938a00a49ebac4117592a43cea7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0ef21938a00a49ebac4117592a43cea7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ef21938a00a49ebac4117592a43cea7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41789 } } }
I20260927 14:51:46.491218 4028 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0ef21938a00a49ebac4117592a43cea7 [sys.catalog]: This master's current role is: LEADER
I20260927 14:51:46.491250 4036 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 14:51:46.494175 4036 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 14:51:46.499935 4036 catalog_manager.cc:1347] Generated new cluster ID: 944a4ac4cebd47abb457e7f44e88a5d8
I20260927 14:51:46.500013 4036 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 14:51:46.514187 4036 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 14:51:46.515507 4036 catalog_manager.cc:1504] Loading token signing keys...
I20260927 14:51:46.536453 4036 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0ef21938a00a49ebac4117592a43cea7: Generated new TSK 0
I20260927 14:51:46.537389 4036 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790520684602251-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790520684602251-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260927 14:51:46.567844 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6936m5/build/debug/bin/kudu
/tmp/dist-test-task6936m5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790520684602251-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790520684602251-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790520684602251-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790520684602251-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41789
--keytab_file=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790520684602251-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40837
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790520684602251-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790520684602251-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790520684602251-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 14:51:46.684120 4048 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 14:51:46.684401 4048 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 14:51:46.684511 4048 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 14:51:46.688546 4048 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 14:51:46.688628 4048 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 14:51:46.688714 4048 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 14:51:46.694147 4048 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40837
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790520684602251-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790520684602251-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790520684602251-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790520684602251-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41789
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4048
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790520684602251-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:15:17 UTC on ccycloud.quasar-ledxfe.root.comops.site
build id 4676134
I20260927 14:51:46.695164 4048 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 14:51:46.696146 4048 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 14:51:46.705416 4054 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 14:51:46.705505 4048 server_base.cc:1034] running on GCE node
W20260927 14:51:46.705652 4053 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 14:51:46.705757 4056 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 14:51:46.706159 4048 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 14:51:46.706753 4048 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 14:51:46.708015 4048 hybrid_clock.cc:648] HybridClock initialized: now 1790520706707992 us; error 48 us; skew 500 ppm
Sep 27 14:51:46 dist-test-slave-xk1g krb5kdc[3941](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790520706, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 14:51:46.710671 4048 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 14:51:46.711545 4048 webserver.cc:460] Webserver started at http://127.0.98.1:35083/ using document root <none> and password file <none>
I20260927 14:51:46.712069 4048 fs_manager.cc:359] Metadata directory not provided
I20260927 14:51:46.712127 4048 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 14:51:46.712325 4048 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 14:51:46.714349 4048 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790520684602251-392-0/minicluster-data/ts-0/data/instance:
uuid: "9b68ddd071524580a27d65467921413a"
format_stamp: "Formatted at 2026-09-27 14:51:46 on dist-test-slave-xk1g"
server_key: "348e628d63d5f8286e908c29bd567cf0"
server_key_iv: "affe88a2dd21698a1a714e6709864923"
server_key_version: "encryptionkey@0"
I20260927 14:51:46.714881 4048 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790520684602251-392-0/minicluster-data/ts-0/wal/instance:
uuid: "9b68ddd071524580a27d65467921413a"
format_stamp: "Formatted at 2026-09-27 14:51:46 on dist-test-slave-xk1g"
server_key: "348e628d63d5f8286e908c29bd567cf0"
server_key_iv: "affe88a2dd21698a1a714e6709864923"
server_key_version: "encryptionkey@0"
I20260927 14:51:46.718415 4048 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.001s
I20260927 14:51:46.720788 4063 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 14:51:46.721992 4048 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260927 14:51:46.722113 4048 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790520684602251-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790520684602251-392-0/minicluster-data/ts-0/wal
uuid: "9b68ddd071524580a27d65467921413a"
format_stamp: "Formatted at 2026-09-27 14:51:46 on dist-test-slave-xk1g"
server_key: "348e628d63d5f8286e908c29bd567cf0"
server_key_iv: "affe88a2dd21698a1a714e6709864923"
server_key_version: "encryptionkey@0"
I20260927 14:51:46.722210 4048 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790520684602251-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790520684602251-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790520684602251-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 14:51:46.731876 4048 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 14:51:46.735195 4048 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 14:51:46.735387 4048 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 14:51:46.735984 4048 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 14:51:46.736774 4048 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 14:51:46.736820 4048 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 14:51:46.736881 4048 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 14:51:46.736912 4048 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 14:51:46.747481 4048 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:46039
I20260927 14:51:46.747551 4176 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:46039 every 8 connection(s)
I20260927 14:51:46.748381 4048 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790520684602251-392-0/minicluster-data/ts-0/data/info.pb
I20260927 14:51:46.754232 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6936m5/build/debug/bin/kudu as pid 4048
I20260927 14:51:46.754359 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790520684602251-392-0/minicluster-data/ts-0/wal/instance
I20260927 14:51:46.754640 392 external_mini_cluster.cc:1428] Setting key 1ea448a749ffd20244baa603977c56da
Sep 27 14:51:46 dist-test-slave-xk1g krb5kdc[3941](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790520706, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 14:51:46.764178 4025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:51:46.750362 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:51031 (local address 127.0.98.62:41789)
0927 14:51:46.750530 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0927 14:51:46.750536 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 14:51:46.751434 (+ 898us) server_negotiation.cc:407] Connection header received
0927 14:51:46.753598 (+ 2164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:51:46.753601 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:51:46.753647 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:51:46.753722 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:51:46.755438 (+ 1716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:51:46.756160 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:51:46.757369 (+ 1209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:51:46.757615 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:51:46.757726 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:51:46.760513 (+ 2787us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:51:46.760517 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:51:46.760520 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 14:51:46.760546 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:51:46.762146 (+ 1600us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:51:46.762666 (+ 520us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:51:46.762671 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:51:46.762673 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:51:46.762753 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:51:46.763097 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:51:46.763102 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:51:46.763104 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:51:46.763280 (+ 176us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:51:46.763384 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0927 14:51:46.763881 (+ 497us) server_negotiation.cc:299] Negotiation successful
0927 14:51:46.764000 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260927 14:51:46.765043 4179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:51:46.750731 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41789 (local address 127.0.98.1:51031)
0927 14:51:46.751237 (+ 506us) negotiation.cc:107] Waiting for socket to connect
0927 14:51:46.751298 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0927 14:51:46.753317 (+ 2019us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:51:46.753954 (+ 637us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:51:46.753966 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:51:46.754529 (+ 563us) client_negotiation.cc:189] Negotiated authn=SASL
0927 14:51:46.755264 (+ 735us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:51:46.755287 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:51:46.756282 (+ 995us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:51:46.756285 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:51:46.757251 (+ 966us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:51:46.757260 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:51:46.757759 (+ 499us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:51:46.757763 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:51:46.757977 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:51:46.758749 (+ 772us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 14:51:46.758775 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0927 14:51:46.760349 (+ 1574us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 14:51:46.762282 (+ 1933us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 14:51:46.762287 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 14:51:46.762302 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0927 14:51:46.762564 (+ 262us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 14:51:46.762846 (+ 282us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 14:51:46.762850 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 14:51:46.762852 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 14:51:46.762992 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 14:51:46.763394 (+ 402us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 14:51:46.763402 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 14:51:46.763654 (+ 252us) client_negotiation.cc:769] Sending connection context
0927 14:51:46.763843 (+ 189us) client_negotiation.cc:240] Negotiation successful
0927 14:51:46.764159 (+ 316us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":251,"thread_start_us":99,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260927 14:51:46.766268 4177 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41789
I20260927 14:51:46.766484 4177 heartbeater.cc:464] Registering TS with master...
I20260927 14:51:46.766963 4177 heartbeater.cc:511] Master 127.0.98.62:41789 requested a full tablet report, sending...
I20260927 14:51:46.768579 3986 ts_manager.cc:194] Registered new tserver with Master: 9b68ddd071524580a27d65467921413a (127.0.98.1:46039)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260927 14:51:46.769856 3986 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:51031
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790520684602251-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790520684602251-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260927 14:51:46.788506 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6936m5/build/debug/bin/kudu
/tmp/dist-test-task6936m5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790520684602251-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790520684602251-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790520684602251-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790520684602251-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41789
--keytab_file=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790520684602251-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40837
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790520684602251-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790520684602251-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790520684602251-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 14:51:46.920771 4182 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 14:51:46.921036 4182 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 14:51:46.921170 4182 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 14:51:46.925437 4182 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 14:51:46.925501 4182 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 14:51:46.925608 4182 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 14:51:46.931021 4182 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40837
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790520684602251-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790520684602251-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790520684602251-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790520684602251-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41789
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4182
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790520684602251-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:15:17 UTC on ccycloud.quasar-ledxfe.root.comops.site
build id 4676134
I20260927 14:51:46.932200 4182 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 14:51:46.933499 4182 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 14:51:46.943678 4187 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 14:51:46.943665 4190 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 14:51:46.943668 4188 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 14:51:46.943779 4182 server_base.cc:1034] running on GCE node
I20260927 14:51:46.944388 4182 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 14:51:46.945181 4182 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 14:51:46.946373 4182 hybrid_clock.cc:648] HybridClock initialized: now 1790520706946361 us; error 40 us; skew 500 ppm
Sep 27 14:51:46 dist-test-slave-xk1g krb5kdc[3941](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790520706, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 14:51:46.949350 4182 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 14:51:46.950426 4182 webserver.cc:460] Webserver started at http://127.0.98.2:33469/ using document root <none> and password file <none>
I20260927 14:51:46.951033 4182 fs_manager.cc:359] Metadata directory not provided
I20260927 14:51:46.951125 4182 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 14:51:46.951365 4182 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 14:51:46.953600 4182 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790520684602251-392-0/minicluster-data/ts-1/data/instance:
uuid: "db1e7a0cbaab41e8869b93ae586b8e94"
format_stamp: "Formatted at 2026-09-27 14:51:46 on dist-test-slave-xk1g"
server_key: "823f5bd5f534435729243ea4572752ae"
server_key_iv: "c9353b5ab6b13546a982007914336fe9"
server_key_version: "encryptionkey@0"
I20260927 14:51:46.954206 4182 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790520684602251-392-0/minicluster-data/ts-1/wal/instance:
uuid: "db1e7a0cbaab41e8869b93ae586b8e94"
format_stamp: "Formatted at 2026-09-27 14:51:46 on dist-test-slave-xk1g"
server_key: "823f5bd5f534435729243ea4572752ae"
server_key_iv: "c9353b5ab6b13546a982007914336fe9"
server_key_version: "encryptionkey@0"
I20260927 14:51:46.958426 4182 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260927 14:51:46.960927 4197 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 14:51:46.962218 4182 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 14:51:46.962352 4182 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790520684602251-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790520684602251-392-0/minicluster-data/ts-1/wal
uuid: "db1e7a0cbaab41e8869b93ae586b8e94"
format_stamp: "Formatted at 2026-09-27 14:51:46 on dist-test-slave-xk1g"
server_key: "823f5bd5f534435729243ea4572752ae"
server_key_iv: "c9353b5ab6b13546a982007914336fe9"
server_key_version: "encryptionkey@0"
I20260927 14:51:46.962466 4182 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790520684602251-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790520684602251-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790520684602251-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 14:51:46.983492 4182 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 14:51:46.986989 4182 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 14:51:46.987196 4182 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 14:51:46.987798 4182 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 14:51:46.988643 4182 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 14:51:46.988716 4182 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 14:51:46.988785 4182 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 14:51:46.988840 4182 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 14:51:46.998240 4182 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:37159
I20260927 14:51:46.998260 4310 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:37159 every 8 connection(s)
I20260927 14:51:46.999409 4182 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790520684602251-392-0/minicluster-data/ts-1/data/info.pb
I20260927 14:51:47.006067 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6936m5/build/debug/bin/kudu as pid 4182
I20260927 14:51:47.006184 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790520684602251-392-0/minicluster-data/ts-1/wal/instance
I20260927 14:51:47.006474 392 external_mini_cluster.cc:1428] Setting key a81571ffdf1e697d030e148e7d0d7884
Sep 27 14:51:47 dist-test-slave-xk1g krb5kdc[3941](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790520706, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 14:51:47.014621 4025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:51:47.001452 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:52629 (local address 127.0.98.62:41789)
0927 14:51:47.001616 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0927 14:51:47.001621 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 14:51:47.002581 (+ 960us) server_negotiation.cc:407] Connection header received
0927 14:51:47.003995 (+ 1414us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:51:47.003998 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:51:47.004048 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:51:47.004130 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:51:47.005463 (+ 1333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:51:47.006179 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:51:47.007493 (+ 1314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:51:47.007803 (+ 310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:51:47.007922 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:51:47.010519 (+ 2597us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:51:47.010523 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:51:47.010525 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 14:51:47.010549 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:51:47.012122 (+ 1573us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:51:47.012802 (+ 680us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:51:47.012805 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:51:47.012807 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:51:47.012895 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:51:47.013381 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:51:47.013384 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:51:47.013385 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:51:47.013572 (+ 187us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:51:47.013660 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0927 14:51:47.014385 (+ 725us) server_negotiation.cc:299] Negotiation successful
0927 14:51:47.014496 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260927 14:51:47.015918 4313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:51:47.001928 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41789 (local address 127.0.98.2:52629)
0927 14:51:47.002400 (+ 472us) negotiation.cc:107] Waiting for socket to connect
0927 14:51:47.002445 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0927 14:51:47.003739 (+ 1294us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:51:47.004299 (+ 560us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:51:47.004305 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:51:47.004684 (+ 379us) client_negotiation.cc:189] Negotiated authn=SASL
0927 14:51:47.005299 (+ 615us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:51:47.005316 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:51:47.006333 (+ 1017us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:51:47.006336 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:51:47.007354 (+ 1018us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:51:47.007364 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:51:47.007975 (+ 611us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:51:47.007979 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:51:47.008153 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:51:47.008797 (+ 644us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 14:51:47.008813 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 14:51:47.010335 (+ 1522us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 14:51:47.012242 (+ 1907us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 14:51:47.012250 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 14:51:47.012270 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0927 14:51:47.012674 (+ 404us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 14:51:47.012999 (+ 325us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 14:51:47.013005 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 14:51:47.013008 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 14:51:47.013247 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 14:51:47.013686 (+ 439us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 14:51:47.013696 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 14:51:47.014064 (+ 368us) client_negotiation.cc:769] Sending connection context
0927 14:51:47.014291 (+ 227us) client_negotiation.cc:240] Negotiation successful
0927 14:51:47.014695 (+ 404us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":274,"thread_start_us":111,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260927 14:51:47.017334 4311 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41789
I20260927 14:51:47.017614 4311 heartbeater.cc:464] Registering TS with master...
I20260927 14:51:47.018137 4311 heartbeater.cc:511] Master 127.0.98.62:41789 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260927 14:51:47.019328 3986 ts_manager.cc:194] Registered new tserver with Master: db1e7a0cbaab41e8869b93ae586b8e94 (127.0.98.2:37159)
I20260927 14:51:47.019938 3986 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:52629
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790520684602251-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790520684602251-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260927 14:51:47.037773 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6936m5/build/debug/bin/kudu
/tmp/dist-test-task6936m5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790520684602251-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790520684602251-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790520684602251-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790520684602251-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41789
--keytab_file=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790520684602251-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40837
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790520684602251-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790520684602251-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790520684602251-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 14:51:47.147042 4316 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 14:51:47.147279 4316 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 14:51:47.147320 4316 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 14:51:47.151358 4316 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 14:51:47.151417 4316 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 14:51:47.151505 4316 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 14:51:47.156774 4316 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40837
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790520684602251-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790520684602251-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790520684602251-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790520684602251-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41789
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4316
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790520684602251-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:15:17 UTC on ccycloud.quasar-ledxfe.root.comops.site
build id 4676134
I20260927 14:51:47.157805 4316 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 14:51:47.158790 4316 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 14:51:47.168069 4322 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 14:51:47.168104 4321 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 14:51:47.168078 4324 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 14:51:47.168305 4316 server_base.cc:1034] running on GCE node
I20260927 14:51:47.168694 4316 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 14:51:47.169363 4316 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 14:51:47.170547 4316 hybrid_clock.cc:648] HybridClock initialized: now 1790520707170528 us; error 25 us; skew 500 ppm
Sep 27 14:51:47 dist-test-slave-xk1g krb5kdc[3941](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790520707, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 14:51:47.173588 4316 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 14:51:47.174427 4316 webserver.cc:460] Webserver started at http://127.0.98.3:38453/ using document root <none> and password file <none>
I20260927 14:51:47.174932 4316 fs_manager.cc:359] Metadata directory not provided
I20260927 14:51:47.174986 4316 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 14:51:47.175145 4316 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 14:51:47.177156 4316 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790520684602251-392-0/minicluster-data/ts-2/data/instance:
uuid: "3aee9d1b9a6743069839aed5f86b6c3f"
format_stamp: "Formatted at 2026-09-27 14:51:47 on dist-test-slave-xk1g"
server_key: "52b7858ee5a5078e0b3554baff852f8d"
server_key_iv: "46ff5e60700a12ee8889aece363a4877"
server_key_version: "encryptionkey@0"
I20260927 14:51:47.177659 4316 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790520684602251-392-0/minicluster-data/ts-2/wal/instance:
uuid: "3aee9d1b9a6743069839aed5f86b6c3f"
format_stamp: "Formatted at 2026-09-27 14:51:47 on dist-test-slave-xk1g"
server_key: "52b7858ee5a5078e0b3554baff852f8d"
server_key_iv: "46ff5e60700a12ee8889aece363a4877"
server_key_version: "encryptionkey@0"
I20260927 14:51:47.181221 4316 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.002s
I20260927 14:51:47.183491 4331 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 14:51:47.184681 4316 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 14:51:47.184810 4316 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790520684602251-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790520684602251-392-0/minicluster-data/ts-2/wal
uuid: "3aee9d1b9a6743069839aed5f86b6c3f"
format_stamp: "Formatted at 2026-09-27 14:51:47 on dist-test-slave-xk1g"
server_key: "52b7858ee5a5078e0b3554baff852f8d"
server_key_iv: "46ff5e60700a12ee8889aece363a4877"
server_key_version: "encryptionkey@0"
I20260927 14:51:47.184921 4316 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790520684602251-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790520684602251-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790520684602251-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 14:51:47.201179 4316 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 14:51:47.204506 4316 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 14:51:47.204704 4316 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 14:51:47.205330 4316 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 14:51:47.206118 4316 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 14:51:47.206183 4316 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 14:51:47.206249 4316 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 14:51:47.206293 4316 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 14:51:47.214679 4316 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:35745
I20260927 14:51:47.214731 4444 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:35745 every 8 connection(s)
I20260927 14:51:47.215606 4316 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790520684602251-392-0/minicluster-data/ts-2/data/info.pb
I20260927 14:51:47.224113 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6936m5/build/debug/bin/kudu as pid 4316
I20260927 14:51:47.224257 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790520684602251-392-0/minicluster-data/ts-2/wal/instance
I20260927 14:51:47.224545 392 external_mini_cluster.cc:1428] Setting key 789dafa4cf8f2da4211f7e90d5af05a7
Sep 27 14:51:47 dist-test-slave-xk1g krb5kdc[3941](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790520707, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 14:51:47.229969 4025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:51:47.217583 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:38729 (local address 127.0.98.62:41789)
0927 14:51:47.217737 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0927 14:51:47.217741 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 14:51:47.218599 (+ 858us) server_negotiation.cc:407] Connection header received
0927 14:51:47.220043 (+ 1444us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:51:47.220047 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:51:47.220101 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:51:47.220182 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:51:47.221484 (+ 1302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:51:47.222017 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:51:47.223170 (+ 1153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:51:47.223395 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:51:47.223504 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:51:47.226638 (+ 3134us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:51:47.226644 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:51:47.226647 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 14:51:47.226676 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:51:47.228109 (+ 1433us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:51:47.228574 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:51:47.228576 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:51:47.228577 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:51:47.228645 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:51:47.228965 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:51:47.228968 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:51:47.228969 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:51:47.229224 (+ 255us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:51:47.229314 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0927 14:51:47.229748 (+ 434us) server_negotiation.cc:299] Negotiation successful
0927 14:51:47.229853 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260927 14:51:47.230794 4447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:51:47.217991 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41789 (local address 127.0.98.3:38729)
0927 14:51:47.218444 (+ 453us) negotiation.cc:107] Waiting for socket to connect
0927 14:51:47.218489 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0927 14:51:47.219810 (+ 1321us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:51:47.220354 (+ 544us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:51:47.220362 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:51:47.220723 (+ 361us) client_negotiation.cc:189] Negotiated authn=SASL
0927 14:51:47.221333 (+ 610us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:51:47.221348 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:51:47.222144 (+ 796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:51:47.222149 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:51:47.223041 (+ 892us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:51:47.223050 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:51:47.223546 (+ 496us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:51:47.223550 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:51:47.223731 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:51:47.224468 (+ 737us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 14:51:47.224485 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 14:51:47.226463 (+ 1978us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 14:51:47.228213 (+ 1750us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 14:51:47.228217 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 14:51:47.228230 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0927 14:51:47.228490 (+ 260us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 14:51:47.228722 (+ 232us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 14:51:47.228724 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 14:51:47.228726 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 14:51:47.228854 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 14:51:47.229311 (+ 457us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 14:51:47.229317 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 14:51:47.229547 (+ 230us) client_negotiation.cc:769] Sending connection context
0927 14:51:47.229703 (+ 156us) client_negotiation.cc:240] Negotiation successful
0927 14:51:47.229979 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":251,"thread_start_us":105,"threads_started":1}
I20260927 14:51:47.231895 4445 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41789
I20260927 14:51:47.232177 4445 heartbeater.cc:464] Registering TS with master...
I20260927 14:51:47.232741 4445 heartbeater.cc:511] Master 127.0.98.62:41789 requested a full tablet report, sending...
I20260927 14:51:47.233847 3986 ts_manager.cc:194] Registered new tserver with Master: 3aee9d1b9a6743069839aed5f86b6c3f (127.0.98.3:35745)
I20260927 14:51:47.234431 3986 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:38729
I20260927 14:51:47.238618 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 27 14:51:47 dist-test-slave-xk1g krb5kdc[3941](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790520707, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 27 14:51:47 dist-test-slave-xk1g krb5kdc[3941](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790520707, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260927 14:51:47.264484 4452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:51:47.254748 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54488 (local address 127.0.98.1:46039)
0927 14:51:47.254994 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0927 14:51:47.255013 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0927 14:51:47.255072 (+ 59us) server_negotiation.cc:407] Connection header received
0927 14:51:47.255127 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:51:47.255132 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:51:47.255327 (+ 195us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:51:47.255450 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:51:47.256125 (+ 675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:51:47.256948 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:51:47.257836 (+ 888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:51:47.258219 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:51:47.258306 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:51:47.260824 (+ 2518us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:51:47.260833 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:51:47.260847 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 14:51:47.260872 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:51:47.262596 (+ 1724us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:51:47.263044 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:51:47.263049 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:51:47.263051 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:51:47.263134 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:51:47.263451 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:51:47.263454 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:51:47.263455 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:51:47.263712 (+ 257us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:51:47.263888 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0927 14:51:47.264100 (+ 212us) server_negotiation.cc:299] Negotiation successful
0927 14:51:47.264267 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":146,"thread_start_us":80,"threads_started":1}
Sep 27 14:51:47 dist-test-slave-xk1g krb5kdc[3941](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790520707, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 14:51:47.275319 4025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:51:47.267597 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34586 (local address 127.0.98.62:41789)
0927 14:51:47.267741 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0927 14:51:47.267744 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 14:51:47.267772 (+ 28us) server_negotiation.cc:407] Connection header received
0927 14:51:47.267887 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:51:47.267889 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:51:47.267929 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:51:47.267999 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:51:47.268565 (+ 566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:51:47.269111 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:51:47.269865 (+ 754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:51:47.270071 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:51:47.270143 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:51:47.272481 (+ 2338us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:51:47.272484 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:51:47.272486 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 14:51:47.272509 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:51:47.273855 (+ 1346us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:51:47.274221 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:51:47.274223 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:51:47.274225 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:51:47.274293 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:51:47.274526 (+ 233us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:51:47.274528 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:51:47.274529 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:51:47.274723 (+ 194us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:51:47.274824 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0927 14:51:47.275028 (+ 204us) server_negotiation.cc:299] Negotiation successful
0927 14:51:47.275115 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
W20260927 14:51:47.275723 3986 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34586
I20260927 14:51:47.276326 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6936m5/build/debug/bin/kudu with pid 4048
I20260927 14:51:47.281703 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6936m5/build/debug/bin/kudu with pid 4182
I20260927 14:51:47.286680 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6936m5/build/debug/bin/kudu with pid 4316
I20260927 14:51:47.292092 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6936m5/build/debug/bin/kudu with pid 3954
2026-09-27T14:51:47Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (2172 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790520684602251-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 14:51:47 dist-test-slave-xk1g krb5kdc[4457](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 14:51:47 dist-test-slave-xk1g krb5kdc[4457](info): set up 2 sockets
Sep 27 14:51:47 dist-test-slave-xk1g krb5kdc[4457](info): commencing operation
krb5kdc: starting...
W20260927 14:51:48.345211 392 mini_kdc.cc:121] Time spent starting KDC: real 1.031s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790520684602251-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790520684602251-392-0/krb5kdc/test-user.keytab.
Sep 27 14:51:48 dist-test-slave-xk1g krb5kdc[4457](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790520708, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T14:51:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T14:51:48Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790520684602251-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790520684602251-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 14:51:48.469996 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6936m5/build/debug/bin/kudu
/tmp/dist-test-task6936m5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790520684602251-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790520684602251-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790520684602251-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790520684602251-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36727
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790520684602251-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44115
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:36727
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790520684602251-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790520684602251-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790520684602251-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 14:51:48.580453 4470 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 14:51:48.580732 4470 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 14:51:48.580777 4470 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 14:51:48.584777 4470 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 14:51:48.584851 4470 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 14:51:48.584874 4470 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 14:51:48.584895 4470 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 14:51:48.584913 4470 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 14:51:48.590086 4470 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44115
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790520684602251-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790520684602251-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:36727
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790520684602251-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36727
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790520684602251-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4470
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790520684602251-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:15:17 UTC on ccycloud.quasar-ledxfe.root.comops.site
build id 4676134
I20260927 14:51:48.591089 4470 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 14:51:48.592084 4470 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 14:51:48.600564 4470 server_base.cc:1034] running on GCE node
W20260927 14:51:48.600518 4475 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 14:51:48.600517 4476 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 14:51:48.600517 4478 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 14:51:48.601398 4470 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 14:51:48.602437 4470 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 14:51:48.603636 4470 hybrid_clock.cc:648] HybridClock initialized: now 1790520708603601 us; error 46 us; skew 500 ppm
Sep 27 14:51:48 dist-test-slave-xk1g krb5kdc[4457](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790520708, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 14:51:48.606591 4470 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 14:51:48.607431 4470 fs_manager.cc:359] Metadata directory not provided
I20260927 14:51:48.607530 4470 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 14:51:48.607754 4470 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 14:51:48.609901 4470 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790520684602251-392-0/minicluster-data/master-0/data/instance:
uuid: "713317bf382b4c8289e4b49012cb3d2d"
format_stamp: "Formatted at 2026-09-27 14:51:48 on dist-test-slave-xk1g"
server_key: "ecc48d42fa7cefddd31b4ab5fb4057a8"
server_key_iv: "da89f94be52607c876c2694dea0a7093"
server_key_version: "encryptionkey@0"
I20260927 14:51:48.610483 4470 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790520684602251-392-0/minicluster-data/master-0/wal/instance:
uuid: "713317bf382b4c8289e4b49012cb3d2d"
format_stamp: "Formatted at 2026-09-27 14:51:48 on dist-test-slave-xk1g"
server_key: "ecc48d42fa7cefddd31b4ab5fb4057a8"
server_key_iv: "da89f94be52607c876c2694dea0a7093"
server_key_version: "encryptionkey@0"
I20260927 14:51:48.614207 4470 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.004s
I20260927 14:51:48.616642 4484 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 14:51:48.617969 4470 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 14:51:48.618129 4470 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790520684602251-392-0/minicluster-data/master-0/data,/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790520684602251-392-0/minicluster-data/master-0/wal
uuid: "713317bf382b4c8289e4b49012cb3d2d"
format_stamp: "Formatted at 2026-09-27 14:51:48 on dist-test-slave-xk1g"
server_key: "ecc48d42fa7cefddd31b4ab5fb4057a8"
server_key_iv: "da89f94be52607c876c2694dea0a7093"
server_key_version: "encryptionkey@0"
I20260927 14:51:48.618247 4470 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790520684602251-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790520684602251-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790520684602251-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 14:51:48.639045 4470 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 14:51:48.641996 4470 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 14:51:48.642220 4470 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 14:51:48.649425 4470 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:36727
I20260927 14:51:48.649744 4536 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:36727 every 8 connection(s)
I20260927 14:51:48.650122 4470 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790520684602251-392-0/minicluster-data/master-0/data/info.pb
I20260927 14:51:48.652688 4537 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 14:51:48.656152 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6936m5/build/debug/bin/kudu as pid 4470
I20260927 14:51:48.656304 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790520684602251-392-0/minicluster-data/master-0/wal/instance
I20260927 14:51:48.656566 392 external_mini_cluster.cc:1428] Setting key c6eea768d056c5f7f931609fd16a7d82
I20260927 14:51:48.658432 4537 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 713317bf382b4c8289e4b49012cb3d2d: Bootstrap starting.
I20260927 14:51:48.660876 4537 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 713317bf382b4c8289e4b49012cb3d2d: Neither blocks nor log segments found. Creating new log.
I20260927 14:51:48.661698 4537 log.cc:824] T 00000000000000000000000000000000 P 713317bf382b4c8289e4b49012cb3d2d: Log is configured to *not* fsync() on all Append() calls
I20260927 14:51:48.663787 4537 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 713317bf382b4c8289e4b49012cb3d2d: No bootstrap required, opened a new log
Sep 27 14:51:48 dist-test-slave-xk1g krb5kdc[4457](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790520708, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 14:51:48.667403 4537 raft_consensus.cc:348] T 00000000000000000000000000000000 P 713317bf382b4c8289e4b49012cb3d2d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "713317bf382b4c8289e4b49012cb3d2d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36727 } }
I20260927 14:51:48.667606 4537 raft_consensus.cc:374] T 00000000000000000000000000000000 P 713317bf382b4c8289e4b49012cb3d2d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 14:51:48.667665 4537 raft_consensus.cc:729] T 00000000000000000000000000000000 P 713317bf382b4c8289e4b49012cb3d2d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 713317bf382b4c8289e4b49012cb3d2d, State: Initialized, Role: FOLLOWER
I20260927 14:51:48.668098 4537 consensus_queue.cc:260] T 00000000000000000000000000000000 P 713317bf382b4c8289e4b49012cb3d2d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "713317bf382b4c8289e4b49012cb3d2d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36727 } }
I20260927 14:51:48.668223 4537 raft_consensus.cc:388] T 00000000000000000000000000000000 P 713317bf382b4c8289e4b49012cb3d2d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 14:51:48.668330 4537 raft_consensus.cc:482] T 00000000000000000000000000000000 P 713317bf382b4c8289e4b49012cb3d2d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 14:51:48.668471 4537 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 713317bf382b4c8289e4b49012cb3d2d [term 0 FOLLOWER]: Advancing to term 1
I20260927 14:51:48.669625 4537 raft_consensus.cc:504] T 00000000000000000000000000000000 P 713317bf382b4c8289e4b49012cb3d2d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "713317bf382b4c8289e4b49012cb3d2d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36727 } }
I20260927 14:51:48.670042 4537 leader_election.cc:302] T 00000000000000000000000000000000 P 713317bf382b4c8289e4b49012cb3d2d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 713317bf382b4c8289e4b49012cb3d2d; no voters:
I20260927 14:51:48.670388 4537 leader_election.cc:288] T 00000000000000000000000000000000 P 713317bf382b4c8289e4b49012cb3d2d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 14:51:48.670482 4542 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 713317bf382b4c8289e4b49012cb3d2d [term 1 FOLLOWER]: Leader election won for term 1
I20260927 14:51:48.670719 4542 raft_consensus.cc:686] T 00000000000000000000000000000000 P 713317bf382b4c8289e4b49012cb3d2d [term 1 LEADER]: Becoming Leader. State: Replica: 713317bf382b4c8289e4b49012cb3d2d, State: Running, Role: LEADER
I20260927 14:51:48.671072 4542 consensus_queue.cc:237] T 00000000000000000000000000000000 P 713317bf382b4c8289e4b49012cb3d2d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "713317bf382b4c8289e4b49012cb3d2d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36727 } }
I20260927 14:51:48.671208 4540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:51:48.657920 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52602 (local address 127.0.98.62:36727)
0927 14:51:48.658305 (+ 385us) server_negotiation.cc:206] Beginning negotiation
0927 14:51:48.658329 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0927 14:51:48.658368 (+ 39us) server_negotiation.cc:407] Connection header received
0927 14:51:48.659007 (+ 639us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:51:48.659023 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:51:48.659348 (+ 325us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:51:48.659603 (+ 255us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:51:48.660504 (+ 901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:51:48.661577 (+ 1073us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:51:48.662592 (+ 1015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:51:48.663082 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:51:48.663204 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:51:48.665890 (+ 2686us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:51:48.665895 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:51:48.665908 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 14:51:48.665932 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:51:48.667616 (+ 1684us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:51:48.668080 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:51:48.668086 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:51:48.668091 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:51:48.668230 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:51:48.668526 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:51:48.668530 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:51:48.668532 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:51:48.669015 (+ 483us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:51:48.669255 (+ 240us) server_negotiation.cc:1035] Waiting for connection context
0927 14:51:48.669757 (+ 502us) server_negotiation.cc:299] Negotiation successful
0927 14:51:48.670115 (+ 358us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":256,"thread_start_us":131,"threads_started":1}
I20260927 14:51:48.671613 4537 sys_catalog.cc:564] T 00000000000000000000000000000000 P 713317bf382b4c8289e4b49012cb3d2d [sys.catalog]: configured and running, proceeding with master startup.
I20260927 14:51:48.672287 4543 sys_catalog.cc:455] T 00000000000000000000000000000000 P 713317bf382b4c8289e4b49012cb3d2d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 713317bf382b4c8289e4b49012cb3d2d. Latest consensus state: current_term: 1 leader_uuid: "713317bf382b4c8289e4b49012cb3d2d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "713317bf382b4c8289e4b49012cb3d2d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36727 } } }
I20260927 14:51:48.672427 4543 sys_catalog.cc:458] T 00000000000000000000000000000000 P 713317bf382b4c8289e4b49012cb3d2d [sys.catalog]: This master's current role is: LEADER
I20260927 14:51:48.672744 4544 sys_catalog.cc:455] T 00000000000000000000000000000000 P 713317bf382b4c8289e4b49012cb3d2d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "713317bf382b4c8289e4b49012cb3d2d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "713317bf382b4c8289e4b49012cb3d2d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36727 } } }
I20260927 14:51:48.672823 4544 sys_catalog.cc:458] T 00000000000000000000000000000000 P 713317bf382b4c8289e4b49012cb3d2d [sys.catalog]: This master's current role is: LEADER
I20260927 14:51:48.674325 4550 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 14:51:48.676816 4550 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 14:51:48.683132 4550 catalog_manager.cc:1347] Generated new cluster ID: e0f2f806d8ab4838a879f207f5554fc7
I20260927 14:51:48.683231 4550 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 14:51:48.700697 4550 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 14:51:48.702003 4550 catalog_manager.cc:1504] Loading token signing keys...
I20260927 14:51:48.712782 4550 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 713317bf382b4c8289e4b49012cb3d2d: Generated new TSK 0
I20260927 14:51:48.713676 4550 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790520684602251-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790520684602251-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260927 14:51:48.748288 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6936m5/build/debug/bin/kudu
/tmp/dist-test-task6936m5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790520684602251-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790520684602251-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790520684602251-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790520684602251-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36727
--keytab_file=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790520684602251-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44115
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790520684602251-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790520684602251-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790520684602251-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 14:51:48.857640 4563 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 14:51:48.857916 4563 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 14:51:48.857962 4563 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 14:51:48.862112 4563 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 14:51:48.862195 4563 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 14:51:48.862258 4563 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 14:51:48.867816 4563 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44115
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790520684602251-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790520684602251-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790520684602251-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790520684602251-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36727
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4563
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790520684602251-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:15:17 UTC on ccycloud.quasar-ledxfe.root.comops.site
build id 4676134
I20260927 14:51:48.868976 4563 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 14:51:48.870102 4563 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 14:51:48.880381 4569 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 14:51:48.880395 4568 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 14:51:48.880445 4563 server_base.cc:1034] running on GCE node
W20260927 14:51:48.880395 4571 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 14:51:48.881145 4563 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 14:51:48.881853 4563 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 14:51:48.883049 4563 hybrid_clock.cc:648] HybridClock initialized: now 1790520708883028 us; error 36 us; skew 500 ppm
Sep 27 14:51:48 dist-test-slave-xk1g krb5kdc[4457](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790520708, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 14:51:48.886112 4563 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 14:51:48.886945 4563 fs_manager.cc:359] Metadata directory not provided
I20260927 14:51:48.887043 4563 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 14:51:48.887264 4563 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 14:51:48.889458 4563 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790520684602251-392-0/minicluster-data/ts-0/data/instance:
uuid: "da207f36066242e5bd3fcf2eec360572"
format_stamp: "Formatted at 2026-09-27 14:51:48 on dist-test-slave-xk1g"
server_key: "31061fb7153cc1a9ba76c50c53658a30"
server_key_iv: "34c0b9df8b3c4b0a7987c1cab39b1bcb"
server_key_version: "encryptionkey@0"
I20260927 14:51:48.890049 4563 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790520684602251-392-0/minicluster-data/ts-0/wal/instance:
uuid: "da207f36066242e5bd3fcf2eec360572"
format_stamp: "Formatted at 2026-09-27 14:51:48 on dist-test-slave-xk1g"
server_key: "31061fb7153cc1a9ba76c50c53658a30"
server_key_iv: "34c0b9df8b3c4b0a7987c1cab39b1bcb"
server_key_version: "encryptionkey@0"
I20260927 14:51:48.893991 4563 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260927 14:51:48.896476 4577 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 14:51:48.897746 4563 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 14:51:48.897883 4563 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790520684602251-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790520684602251-392-0/minicluster-data/ts-0/wal
uuid: "da207f36066242e5bd3fcf2eec360572"
format_stamp: "Formatted at 2026-09-27 14:51:48 on dist-test-slave-xk1g"
server_key: "31061fb7153cc1a9ba76c50c53658a30"
server_key_iv: "34c0b9df8b3c4b0a7987c1cab39b1bcb"
server_key_version: "encryptionkey@0"
I20260927 14:51:48.898000 4563 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790520684602251-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790520684602251-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790520684602251-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 14:51:48.930337 4563 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 14:51:48.933665 4563 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 14:51:48.933874 4563 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 14:51:48.934504 4563 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 14:51:48.935369 4563 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 14:51:48.935444 4563 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 14:51:48.935524 4563 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 14:51:48.935567 4563 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 14:51:48.944612 4563 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:39317
I20260927 14:51:48.944629 4690 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:39317 every 8 connection(s)
I20260927 14:51:48.945264 4563 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790520684602251-392-0/minicluster-data/ts-0/data/info.pb
I20260927 14:51:48.954507 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6936m5/build/debug/bin/kudu as pid 4563
Sep 27 14:51:48 dist-test-slave-xk1g krb5kdc[4457](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790520708, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 14:51:48.954643 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790520684602251-392-0/minicluster-data/ts-0/wal/instance
I20260927 14:51:48.954902 392 external_mini_cluster.cc:1428] Setting key 1b2c359d3f16eb83905cef26794fa01a
I20260927 14:51:48.958811 4540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:51:48.947049 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:51735 (local address 127.0.98.62:36727)
0927 14:51:48.947184 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0927 14:51:48.947188 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 14:51:48.947970 (+ 782us) server_negotiation.cc:407] Connection header received
0927 14:51:48.949399 (+ 1429us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:51:48.949402 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:51:48.949469 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:51:48.949577 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:51:48.950744 (+ 1167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:51:48.951280 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:51:48.952419 (+ 1139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:51:48.952654 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:51:48.952769 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:51:48.955202 (+ 2433us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:51:48.955205 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:51:48.955207 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 14:51:48.955253 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:51:48.956736 (+ 1483us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:51:48.957311 (+ 575us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:51:48.957316 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:51:48.957318 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:51:48.957397 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:51:48.957750 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:51:48.957752 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:51:48.957754 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:51:48.957927 (+ 173us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:51:48.958034 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0927 14:51:48.958524 (+ 490us) server_negotiation.cc:299] Negotiation successful
0927 14:51:48.958646 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260927 14:51:48.959630 4693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:51:48.947369 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36727 (local address 127.0.98.1:51735)
0927 14:51:48.947817 (+ 448us) negotiation.cc:107] Waiting for socket to connect
0927 14:51:48.947860 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0927 14:51:48.949192 (+ 1332us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:51:48.949754 (+ 562us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:51:48.949762 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:51:48.950123 (+ 361us) client_negotiation.cc:189] Negotiated authn=SASL
0927 14:51:48.950594 (+ 471us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:51:48.950608 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:51:48.951410 (+ 802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:51:48.951413 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:51:48.952287 (+ 874us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:51:48.952296 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:51:48.952806 (+ 510us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:51:48.952810 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:51:48.952991 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:51:48.953561 (+ 570us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 14:51:48.953576 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 14:51:48.955037 (+ 1461us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 14:51:48.956852 (+ 1815us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 14:51:48.956857 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 14:51:48.956872 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0927 14:51:48.957212 (+ 340us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 14:51:48.957507 (+ 295us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 14:51:48.957510 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 14:51:48.957512 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 14:51:48.957650 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 14:51:48.958031 (+ 381us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 14:51:48.958038 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 14:51:48.958284 (+ 246us) client_negotiation.cc:769] Sending connection context
0927 14:51:48.958461 (+ 177us) client_negotiation.cc:240] Negotiation successful
0927 14:51:48.958781 (+ 320us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":269,"thread_start_us":116,"threads_started":1}
I20260927 14:51:48.960995 4691 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36727
I20260927 14:51:48.961268 4691 heartbeater.cc:464] Registering TS with master...
I20260927 14:51:48.961725 4691 heartbeater.cc:511] Master 127.0.98.62:36727 requested a full tablet report, sending...
I20260927 14:51:48.963200 4501 ts_manager.cc:194] Registered new tserver with Master: da207f36066242e5bd3fcf2eec360572 (127.0.98.1:39317)
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260927 14:51:48.964362 4501 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:51735
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790520684602251-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790520684602251-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260927 14:51:48.983992 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6936m5/build/debug/bin/kudu
/tmp/dist-test-task6936m5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790520684602251-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790520684602251-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790520684602251-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790520684602251-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36727
--keytab_file=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790520684602251-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44115
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790520684602251-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790520684602251-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790520684602251-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 14:51:49.092725 4696 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 14:51:49.093031 4696 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 14:51:49.093137 4696 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 14:51:49.097110 4696 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 14:51:49.097199 4696 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 14:51:49.097294 4696 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 14:51:49.102587 4696 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44115
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790520684602251-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790520684602251-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790520684602251-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790520684602251-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36727
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4696
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790520684602251-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:15:17 UTC on ccycloud.quasar-ledxfe.root.comops.site
build id 4676134
I20260927 14:51:49.103626 4696 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 14:51:49.104586 4696 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 14:51:49.113663 4704 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 14:51:49.113663 4701 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 14:51:49.113669 4702 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 14:51:49.113742 4696 server_base.cc:1034] running on GCE node
I20260927 14:51:49.114369 4696 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 14:51:49.114975 4696 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 14:51:49.116168 4696 hybrid_clock.cc:648] HybridClock initialized: now 1790520709116150 us; error 34 us; skew 500 ppm
Sep 27 14:51:49 dist-test-slave-xk1g krb5kdc[4457](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790520709, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 14:51:49.118897 4696 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 14:51:49.119597 4696 fs_manager.cc:359] Metadata directory not provided
I20260927 14:51:49.119684 4696 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 14:51:49.119897 4696 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 14:51:49.121969 4696 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790520684602251-392-0/minicluster-data/ts-1/data/instance:
uuid: "3b962746e285451ca0add90ca3ca2dd4"
format_stamp: "Formatted at 2026-09-27 14:51:49 on dist-test-slave-xk1g"
server_key: "970328b643c86c0247be0dd240ec551b"
server_key_iv: "2b8e96c7a5cd8f3e8d7c0c90d2cedf6f"
server_key_version: "encryptionkey@0"
I20260927 14:51:49.122511 4696 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790520684602251-392-0/minicluster-data/ts-1/wal/instance:
uuid: "3b962746e285451ca0add90ca3ca2dd4"
format_stamp: "Formatted at 2026-09-27 14:51:49 on dist-test-slave-xk1g"
server_key: "970328b643c86c0247be0dd240ec551b"
server_key_iv: "2b8e96c7a5cd8f3e8d7c0c90d2cedf6f"
server_key_version: "encryptionkey@0"
I20260927 14:51:49.126106 4696 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.004s
I20260927 14:51:49.128346 4710 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 14:51:49.129665 4696 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.002s
I20260927 14:51:49.129794 4696 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790520684602251-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790520684602251-392-0/minicluster-data/ts-1/wal
uuid: "3b962746e285451ca0add90ca3ca2dd4"
format_stamp: "Formatted at 2026-09-27 14:51:49 on dist-test-slave-xk1g"
server_key: "970328b643c86c0247be0dd240ec551b"
server_key_iv: "2b8e96c7a5cd8f3e8d7c0c90d2cedf6f"
server_key_version: "encryptionkey@0"
I20260927 14:51:49.129899 4696 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790520684602251-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790520684602251-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790520684602251-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 14:51:49.157778 4696 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 14:51:49.161332 4696 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 14:51:49.161554 4696 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 14:51:49.162168 4696 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 14:51:49.163007 4696 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 14:51:49.163089 4696 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 14:51:49.163163 4696 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 14:51:49.163213 4696 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 14:51:49.172266 4696 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:43821
I20260927 14:51:49.172410 4823 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:43821 every 8 connection(s)
I20260927 14:51:49.173007 4696 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790520684602251-392-0/minicluster-data/ts-1/data/info.pb
I20260927 14:51:49.180528 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6936m5/build/debug/bin/kudu as pid 4696
I20260927 14:51:49.180666 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790520684602251-392-0/minicluster-data/ts-1/wal/instance
I20260927 14:51:49.180962 392 external_mini_cluster.cc:1428] Setting key bd29029c69e246286d9427f86ac67f31
Sep 27 14:51:49 dist-test-slave-xk1g krb5kdc[4457](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790520709, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 14:51:49.188072 4540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:51:49.174917 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:36347 (local address 127.0.98.62:36727)
0927 14:51:49.175053 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0927 14:51:49.175057 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 14:51:49.175860 (+ 803us) server_negotiation.cc:407] Connection header received
0927 14:51:49.177346 (+ 1486us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:51:49.177349 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:51:49.177413 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:51:49.177523 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:51:49.178764 (+ 1241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:51:49.179313 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:51:49.180509 (+ 1196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:51:49.180833 (+ 324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:51:49.180958 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:51:49.183990 (+ 3032us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:51:49.183997 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:51:49.184003 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 14:51:49.184051 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:51:49.185678 (+ 1627us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:51:49.186313 (+ 635us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:51:49.186316 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:51:49.186317 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:51:49.186424 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:51:49.186862 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:51:49.186865 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:51:49.186866 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:51:49.187051 (+ 185us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:51:49.187126 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0927 14:51:49.187840 (+ 714us) server_negotiation.cc:299] Negotiation successful
0927 14:51:49.187941 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260927 14:51:49.189424 4826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:51:49.175229 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36727 (local address 127.0.98.2:36347)
0927 14:51:49.175677 (+ 448us) negotiation.cc:107] Waiting for socket to connect
0927 14:51:49.175722 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0927 14:51:49.177038 (+ 1316us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:51:49.177706 (+ 668us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:51:49.177713 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:51:49.178101 (+ 388us) client_negotiation.cc:189] Negotiated authn=SASL
0927 14:51:49.178604 (+ 503us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:51:49.178618 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:51:49.179451 (+ 833us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:51:49.179456 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:51:49.180374 (+ 918us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:51:49.180384 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:51:49.180997 (+ 613us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:51:49.181002 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:51:49.181218 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:51:49.181940 (+ 722us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 14:51:49.181963 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0927 14:51:49.183810 (+ 1847us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 14:51:49.185797 (+ 1987us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 14:51:49.185807 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 14:51:49.185826 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0927 14:51:49.186214 (+ 388us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 14:51:49.186536 (+ 322us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 14:51:49.186543 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 14:51:49.186547 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 14:51:49.186765 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 14:51:49.187158 (+ 393us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 14:51:49.187169 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 14:51:49.187555 (+ 386us) client_negotiation.cc:769] Sending connection context
0927 14:51:49.187796 (+ 241us) client_negotiation.cc:240] Negotiation successful
0927 14:51:49.188220 (+ 424us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":252,"thread_start_us":97,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260927 14:51:49.190837 4824 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36727
I20260927 14:51:49.191110 4824 heartbeater.cc:464] Registering TS with master...
I20260927 14:51:49.191542 4824 heartbeater.cc:511] Master 127.0.98.62:36727 requested a full tablet report, sending...
I20260927 14:51:49.192682 4501 ts_manager.cc:194] Registered new tserver with Master: 3b962746e285451ca0add90ca3ca2dd4 (127.0.98.2:43821)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260927 14:51:49.193377 4501 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:36347
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790520684602251-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790520684602251-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260927 14:51:49.211923 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6936m5/build/debug/bin/kudu
/tmp/dist-test-task6936m5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790520684602251-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790520684602251-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790520684602251-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790520684602251-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36727
--keytab_file=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790520684602251-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44115
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790520684602251-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790520684602251-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790520684602251-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 14:51:49.323662 4829 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 14:51:49.323925 4829 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 14:51:49.323973 4829 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 14:51:49.328225 4829 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 14:51:49.328289 4829 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 14:51:49.328351 4829 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 14:51:49.333640 4829 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44115
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790520684602251-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790520684602251-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790520684602251-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790520684602251-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36727
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4829
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790520684602251-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:15:17 UTC on ccycloud.quasar-ledxfe.root.comops.site
build id 4676134
I20260927 14:51:49.334673 4829 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 14:51:49.335664 4829 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 14:51:49.344790 4835 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 14:51:49.344790 4834 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 14:51:49.344797 4837 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 14:51:49.345125 4829 server_base.cc:1034] running on GCE node
I20260927 14:51:49.345587 4829 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 14:51:49.346195 4829 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 14:51:49.347393 4829 hybrid_clock.cc:648] HybridClock initialized: now 1790520709347374 us; error 29 us; skew 500 ppm
Sep 27 14:51:49 dist-test-slave-xk1g krb5kdc[4457](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790520709, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 14:51:49.350476 4829 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 14:51:49.351310 4829 fs_manager.cc:359] Metadata directory not provided
I20260927 14:51:49.351423 4829 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 14:51:49.351676 4829 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 14:51:49.353930 4829 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790520684602251-392-0/minicluster-data/ts-2/data/instance:
uuid: "d63f82fe261947e5946447215824a375"
format_stamp: "Formatted at 2026-09-27 14:51:49 on dist-test-slave-xk1g"
server_key: "fd62fcf49f32e72a9614e1bd487baa7a"
server_key_iv: "87cb201272d8619f0925e4555dfecda3"
server_key_version: "encryptionkey@0"
I20260927 14:51:49.354530 4829 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790520684602251-392-0/minicluster-data/ts-2/wal/instance:
uuid: "d63f82fe261947e5946447215824a375"
format_stamp: "Formatted at 2026-09-27 14:51:49 on dist-test-slave-xk1g"
server_key: "fd62fcf49f32e72a9614e1bd487baa7a"
server_key_iv: "87cb201272d8619f0925e4555dfecda3"
server_key_version: "encryptionkey@0"
I20260927 14:51:49.358430 4829 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260927 14:51:49.361014 4843 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 14:51:49.362334 4829 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 14:51:49.362483 4829 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790520684602251-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790520684602251-392-0/minicluster-data/ts-2/wal
uuid: "d63f82fe261947e5946447215824a375"
format_stamp: "Formatted at 2026-09-27 14:51:49 on dist-test-slave-xk1g"
server_key: "fd62fcf49f32e72a9614e1bd487baa7a"
server_key_iv: "87cb201272d8619f0925e4555dfecda3"
server_key_version: "encryptionkey@0"
I20260927 14:51:49.362596 4829 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790520684602251-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790520684602251-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790520684602251-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 14:51:49.373236 4829 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 14:51:49.376399 4829 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 14:51:49.376597 4829 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 14:51:49.377241 4829 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 14:51:49.378047 4829 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 14:51:49.378119 4829 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 14:51:49.378177 4829 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 14:51:49.378228 4829 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 14:51:49.386893 4829 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:35775
I20260927 14:51:49.386909 4956 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:35775 every 8 connection(s)
I20260927 14:51:49.387475 4829 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790520684602251-392-0/minicluster-data/ts-2/data/info.pb
I20260927 14:51:49.388978 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6936m5/build/debug/bin/kudu as pid 4829
I20260927 14:51:49.389155 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790520684602251-392-0/minicluster-data/ts-2/wal/instance
I20260927 14:51:49.389447 392 external_mini_cluster.cc:1428] Setting key d748d6deb518cd00bc3ecb9762518050
Sep 27 14:51:49 dist-test-slave-xk1g krb5kdc[4457](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790520709, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 14:51:49.401093 4540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:51:49.389166 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:57937 (local address 127.0.98.62:36727)
0927 14:51:49.389299 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0927 14:51:49.389303 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 14:51:49.390348 (+ 1045us) server_negotiation.cc:407] Connection header received
0927 14:51:49.391861 (+ 1513us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:51:49.391864 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:51:49.391911 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:51:49.391977 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:51:49.393118 (+ 1141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:51:49.393641 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:51:49.394824 (+ 1183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:51:49.395042 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:51:49.395116 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:51:49.397676 (+ 2560us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:51:49.397680 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:51:49.397683 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 14:51:49.397721 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:51:49.399063 (+ 1342us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:51:49.399581 (+ 518us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:51:49.399585 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:51:49.399587 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:51:49.399673 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:51:49.400011 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:51:49.400016 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:51:49.400018 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:51:49.400195 (+ 177us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:51:49.400300 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0927 14:51:49.400787 (+ 487us) server_negotiation.cc:299] Negotiation successful
0927 14:51:49.400905 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260927 14:51:49.401954 4959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:51:49.389639 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36727 (local address 127.0.98.3:57937)
0927 14:51:49.390194 (+ 555us) negotiation.cc:107] Waiting for socket to connect
0927 14:51:49.390244 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0927 14:51:49.391692 (+ 1448us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:51:49.392113 (+ 421us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:51:49.392119 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:51:49.392460 (+ 341us) client_negotiation.cc:189] Negotiated authn=SASL
0927 14:51:49.392934 (+ 474us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:51:49.392947 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:51:49.393777 (+ 830us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:51:49.393780 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:51:49.394702 (+ 922us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:51:49.394712 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:51:49.395181 (+ 469us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:51:49.395184 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:51:49.395343 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:51:49.395991 (+ 648us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 14:51:49.396009 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0927 14:51:49.397503 (+ 1494us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 14:51:49.399193 (+ 1690us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 14:51:49.399199 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 14:51:49.399213 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0927 14:51:49.399471 (+ 258us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 14:51:49.399765 (+ 294us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 14:51:49.399770 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 14:51:49.399772 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 14:51:49.399914 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 14:51:49.400298 (+ 384us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 14:51:49.400305 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 14:51:49.400547 (+ 242us) client_negotiation.cc:769] Sending connection context
0927 14:51:49.400734 (+ 187us) client_negotiation.cc:240] Negotiation successful
0927 14:51:49.401074 (+ 340us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":329,"thread_start_us":139,"threads_started":1}
I20260927 14:51:49.403229 4957 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36727
I20260927 14:51:49.403488 4957 heartbeater.cc:464] Registering TS with master...
I20260927 14:51:49.403929 4957 heartbeater.cc:511] Master 127.0.98.62:36727 requested a full tablet report, sending...
I20260927 14:51:49.404881 4501 ts_manager.cc:194] Registered new tserver with Master: d63f82fe261947e5946447215824a375 (127.0.98.3:35775)
I20260927 14:51:49.405494 4501 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:57937
I20260927 14:51:49.415091 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 14:51:49.424731 4540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:51:49.417152 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52604 (local address 127.0.98.62:36727)
0927 14:51:49.417280 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0927 14:51:49.417284 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 14:51:49.417389 (+ 105us) server_negotiation.cc:407] Connection header received
0927 14:51:49.417465 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:51:49.417467 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:51:49.417509 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:51:49.417583 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:51:49.418286 (+ 703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:51:49.418805 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:51:49.419634 (+ 829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:51:49.419836 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:51:49.419934 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:51:49.421405 (+ 1471us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:51:49.421407 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:51:49.421409 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 14:51:49.421432 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:51:49.423020 (+ 1588us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:51:49.423439 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:51:49.423442 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:51:49.423443 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:51:49.423517 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:51:49.423749 (+ 232us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:51:49.423751 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:51:49.423752 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:51:49.424033 (+ 281us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:51:49.424139 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0927 14:51:49.424350 (+ 211us) server_negotiation.cc:299] Negotiation successful
0927 14:51:49.424524 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260927 14:51:49.427917 4501 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52604:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 14:51:49.431075 4501 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 14:51:49.445382 4969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:51:49.439521 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:39317 (local address 127.0.0.1:49258)
0927 14:51:49.439814 (+ 293us) negotiation.cc:107] Waiting for socket to connect
0927 14:51:49.439871 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0927 14:51:49.439985 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:51:49.440941 (+ 956us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:51:49.440947 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:51:49.440968 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 14:51:49.441133 (+ 165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:51:49.441147 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:51:49.442437 (+ 1290us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:51:49.442443 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:51:49.443645 (+ 1202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:51:49.443655 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:51:49.444745 (+ 1090us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:51:49.444748 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:51:49.444862 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:51:49.444907 (+ 45us) client_negotiation.cc:769] Sending connection context
0927 14:51:49.445035 (+ 128us) client_negotiation.cc:240] Negotiation successful
0927 14:51:49.445206 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":197,"thread_start_us":127,"threads_started":1}
I20260927 14:51:49.445413 4971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:51:49.440207 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49258 (local address 127.0.98.1:39317)
0927 14:51:49.440503 (+ 296us) server_negotiation.cc:206] Beginning negotiation
0927 14:51:49.440514 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 14:51:49.440540 (+ 26us) server_negotiation.cc:407] Connection header received
0927 14:51:49.440618 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:51:49.440622 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:51:49.440778 (+ 156us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:51:49.440953 (+ 175us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 14:51:49.441295 (+ 342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:51:49.442312 (+ 1017us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:51:49.443783 (+ 1471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:51:49.444615 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:51:49.444724 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:51:49.444891 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0927 14:51:49.445122 (+ 231us) server_negotiation.cc:299] Negotiation successful
0927 14:51:49.445246 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":190,"thread_start_us":62,"threads_started":1}
I20260927 14:51:49.445644 4968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:51:49.439433 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:35775 (local address 127.0.0.1:55302)
0927 14:51:49.439946 (+ 513us) negotiation.cc:107] Waiting for socket to connect
0927 14:51:49.439960 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 14:51:49.440045 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:51:49.441273 (+ 1228us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:51:49.441277 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:51:49.441295 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 14:51:49.441400 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:51:49.441406 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:51:49.443007 (+ 1601us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:51:49.443011 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:51:49.444128 (+ 1117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:51:49.444137 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:51:49.445258 (+ 1121us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:51:49.445261 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:51:49.445368 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:51:49.445390 (+ 22us) client_negotiation.cc:769] Sending connection context
0927 14:51:49.445445 (+ 55us) client_negotiation.cc:240] Negotiation successful
0927 14:51:49.445526 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":157,"mutex_wait_us":2,"thread_start_us":97,"threads_started":1}
I20260927 14:51:49.445802 4970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:51:49.440135 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:43821 (local address 127.0.0.1:55106)
0927 14:51:49.440374 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0927 14:51:49.440388 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 14:51:49.440471 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:51:49.441801 (+ 1330us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:51:49.441804 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:51:49.441821 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 14:51:49.441904 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:51:49.441909 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:51:49.443186 (+ 1277us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:51:49.443191 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:51:49.444196 (+ 1005us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:51:49.444205 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:51:49.445361 (+ 1156us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:51:49.445364 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:51:49.445474 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:51:49.445503 (+ 29us) client_negotiation.cc:769] Sending connection context
0927 14:51:49.445589 (+ 86us) client_negotiation.cc:240] Negotiation successful
0927 14:51:49.445670 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":170,"mutex_wait_us":29,"thread_start_us":107,"threads_started":1}
I20260927 14:51:49.446225 4973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:51:49.440825 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55106 (local address 127.0.98.2:43821)
0927 14:51:49.441368 (+ 543us) server_negotiation.cc:206] Beginning negotiation
0927 14:51:49.441381 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 14:51:49.441401 (+ 20us) server_negotiation.cc:407] Connection header received
0927 14:51:49.441475 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:51:49.441483 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:51:49.441658 (+ 175us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:51:49.441805 (+ 147us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 14:51:49.442001 (+ 196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:51:49.443059 (+ 1058us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:51:49.444329 (+ 1270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:51:49.445078 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:51:49.445168 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:51:49.445301 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0927 14:51:49.445953 (+ 652us) server_negotiation.cc:299] Negotiation successful
0927 14:51:49.446056 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":441,"thread_start_us":88,"threads_started":1}
I20260927 14:51:49.446412 4972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:51:49.440401 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55302 (local address 127.0.98.3:35775)
0927 14:51:49.440740 (+ 339us) server_negotiation.cc:206] Beginning negotiation
0927 14:51:49.440751 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 14:51:49.440775 (+ 24us) server_negotiation.cc:407] Connection header received
0927 14:51:49.440862 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:51:49.440868 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:51:49.441116 (+ 248us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:51:49.441544 (+ 428us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 14:51:49.441691 (+ 147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:51:49.442861 (+ 1170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:51:49.444272 (+ 1411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:51:49.445126 (+ 854us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:51:49.445974 (+ 848us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:51:49.446117 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0927 14:51:49.446169 (+ 52us) server_negotiation.cc:299] Negotiation successful
0927 14:51:49.446257 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":248,"thread_start_us":167,"threads_started":1}
I20260927 14:51:49.448940 4758 tablet_service.cc:1467] Processing CreateTablet for tablet 500815feacff4dc18e98f410ae39777f (DEFAULT_TABLE table=test-table [id=e7638b4dd64f438cbbd01a4ce0f1ffb6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 14:51:49.448944 4625 tablet_service.cc:1467] Processing CreateTablet for tablet 500815feacff4dc18e98f410ae39777f (DEFAULT_TABLE table=test-table [id=e7638b4dd64f438cbbd01a4ce0f1ffb6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 14:51:49.448946 4891 tablet_service.cc:1467] Processing CreateTablet for tablet 500815feacff4dc18e98f410ae39777f (DEFAULT_TABLE table=test-table [id=e7638b4dd64f438cbbd01a4ce0f1ffb6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 14:51:49.450073 4758 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 500815feacff4dc18e98f410ae39777f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 14:51:49.450093 4891 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 500815feacff4dc18e98f410ae39777f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 14:51:49.450086 4625 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 500815feacff4dc18e98f410ae39777f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 14:51:49.456905 4974 tablet_bootstrap.cc:492] T 500815feacff4dc18e98f410ae39777f P 3b962746e285451ca0add90ca3ca2dd4: Bootstrap starting.
I20260927 14:51:49.457931 4975 tablet_bootstrap.cc:492] T 500815feacff4dc18e98f410ae39777f P d63f82fe261947e5946447215824a375: Bootstrap starting.
I20260927 14:51:49.459048 4974 tablet_bootstrap.cc:654] T 500815feacff4dc18e98f410ae39777f P 3b962746e285451ca0add90ca3ca2dd4: Neither blocks nor log segments found. Creating new log.
I20260927 14:51:49.459466 4976 tablet_bootstrap.cc:492] T 500815feacff4dc18e98f410ae39777f P da207f36066242e5bd3fcf2eec360572: Bootstrap starting.
I20260927 14:51:49.459779 4975 tablet_bootstrap.cc:654] T 500815feacff4dc18e98f410ae39777f P d63f82fe261947e5946447215824a375: Neither blocks nor log segments found. Creating new log.
I20260927 14:51:49.460291 4974 log.cc:824] T 500815feacff4dc18e98f410ae39777f P 3b962746e285451ca0add90ca3ca2dd4: Log is configured to *not* fsync() on all Append() calls
I20260927 14:51:49.461028 4975 log.cc:824] T 500815feacff4dc18e98f410ae39777f P d63f82fe261947e5946447215824a375: Log is configured to *not* fsync() on all Append() calls
I20260927 14:51:49.461683 4976 tablet_bootstrap.cc:654] T 500815feacff4dc18e98f410ae39777f P da207f36066242e5bd3fcf2eec360572: Neither blocks nor log segments found. Creating new log.
I20260927 14:51:49.462409 4974 tablet_bootstrap.cc:492] T 500815feacff4dc18e98f410ae39777f P 3b962746e285451ca0add90ca3ca2dd4: No bootstrap required, opened a new log
I20260927 14:51:49.462556 4976 log.cc:824] T 500815feacff4dc18e98f410ae39777f P da207f36066242e5bd3fcf2eec360572: Log is configured to *not* fsync() on all Append() calls
I20260927 14:51:49.462648 4974 ts_tablet_manager.cc:1397] T 500815feacff4dc18e98f410ae39777f P 3b962746e285451ca0add90ca3ca2dd4: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.004s
I20260927 14:51:49.463274 4975 tablet_bootstrap.cc:492] T 500815feacff4dc18e98f410ae39777f P d63f82fe261947e5946447215824a375: No bootstrap required, opened a new log
I20260927 14:51:49.463455 4975 ts_tablet_manager.cc:1397] T 500815feacff4dc18e98f410ae39777f P d63f82fe261947e5946447215824a375: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260927 14:51:49.464359 4976 tablet_bootstrap.cc:492] T 500815feacff4dc18e98f410ae39777f P da207f36066242e5bd3fcf2eec360572: No bootstrap required, opened a new log
I20260927 14:51:49.464519 4976 ts_tablet_manager.cc:1397] T 500815feacff4dc18e98f410ae39777f P da207f36066242e5bd3fcf2eec360572: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260927 14:51:49.465981 4974 raft_consensus.cc:348] T 500815feacff4dc18e98f410ae39777f P 3b962746e285451ca0add90ca3ca2dd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d63f82fe261947e5946447215824a375" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35775 } } peers { permanent_uuid: "da207f36066242e5bd3fcf2eec360572" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39317 } } peers { permanent_uuid: "3b962746e285451ca0add90ca3ca2dd4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43821 } }
I20260927 14:51:49.466318 4974 raft_consensus.cc:374] T 500815feacff4dc18e98f410ae39777f P 3b962746e285451ca0add90ca3ca2dd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 14:51:49.466394 4974 raft_consensus.cc:729] T 500815feacff4dc18e98f410ae39777f P 3b962746e285451ca0add90ca3ca2dd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b962746e285451ca0add90ca3ca2dd4, State: Initialized, Role: FOLLOWER
I20260927 14:51:49.466863 4974 consensus_queue.cc:260] T 500815feacff4dc18e98f410ae39777f P 3b962746e285451ca0add90ca3ca2dd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d63f82fe261947e5946447215824a375" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35775 } } peers { permanent_uuid: "da207f36066242e5bd3fcf2eec360572" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39317 } } peers { permanent_uuid: "3b962746e285451ca0add90ca3ca2dd4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43821 } }
I20260927 14:51:49.467300 4975 raft_consensus.cc:348] T 500815feacff4dc18e98f410ae39777f P d63f82fe261947e5946447215824a375 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d63f82fe261947e5946447215824a375" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35775 } } peers { permanent_uuid: "da207f36066242e5bd3fcf2eec360572" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39317 } } peers { permanent_uuid: "3b962746e285451ca0add90ca3ca2dd4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43821 } }
I20260927 14:51:49.467502 4824 heartbeater.cc:503] Master 127.0.98.62:36727 was elected leader, sending a full tablet report...
I20260927 14:51:49.467612 4975 raft_consensus.cc:374] T 500815feacff4dc18e98f410ae39777f P d63f82fe261947e5946447215824a375 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 14:51:49.467710 4975 raft_consensus.cc:729] T 500815feacff4dc18e98f410ae39777f P d63f82fe261947e5946447215824a375 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d63f82fe261947e5946447215824a375, State: Initialized, Role: FOLLOWER
I20260927 14:51:49.467836 4974 ts_tablet_manager.cc:1428] T 500815feacff4dc18e98f410ae39777f P 3b962746e285451ca0add90ca3ca2dd4: Time spent starting tablet: real 0.005s user 0.003s sys 0.003s
I20260927 14:51:49.468204 4975 consensus_queue.cc:260] T 500815feacff4dc18e98f410ae39777f P d63f82fe261947e5946447215824a375 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d63f82fe261947e5946447215824a375" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35775 } } peers { permanent_uuid: "da207f36066242e5bd3fcf2eec360572" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39317 } } peers { permanent_uuid: "3b962746e285451ca0add90ca3ca2dd4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43821 } }
I20260927 14:51:49.468678 4976 raft_consensus.cc:348] T 500815feacff4dc18e98f410ae39777f P da207f36066242e5bd3fcf2eec360572 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d63f82fe261947e5946447215824a375" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35775 } } peers { permanent_uuid: "da207f36066242e5bd3fcf2eec360572" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39317 } } peers { permanent_uuid: "3b962746e285451ca0add90ca3ca2dd4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43821 } }
I20260927 14:51:49.468966 4976 raft_consensus.cc:374] T 500815feacff4dc18e98f410ae39777f P da207f36066242e5bd3fcf2eec360572 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 14:51:49.469079 4976 raft_consensus.cc:729] T 500815feacff4dc18e98f410ae39777f P da207f36066242e5bd3fcf2eec360572 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: da207f36066242e5bd3fcf2eec360572, State: Initialized, Role: FOLLOWER
I20260927 14:51:49.469264 4975 ts_tablet_manager.cc:1428] T 500815feacff4dc18e98f410ae39777f P d63f82fe261947e5946447215824a375: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260927 14:51:49.469542 4976 consensus_queue.cc:260] T 500815feacff4dc18e98f410ae39777f P da207f36066242e5bd3fcf2eec360572 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d63f82fe261947e5946447215824a375" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35775 } } peers { permanent_uuid: "da207f36066242e5bd3fcf2eec360572" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39317 } } peers { permanent_uuid: "3b962746e285451ca0add90ca3ca2dd4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43821 } }
I20260927 14:51:49.469599 4957 heartbeater.cc:503] Master 127.0.98.62:36727 was elected leader, sending a full tablet report...
I20260927 14:51:49.470146 4691 heartbeater.cc:503] Master 127.0.98.62:36727 was elected leader, sending a full tablet report...
I20260927 14:51:49.470459 4976 ts_tablet_manager.cc:1428] T 500815feacff4dc18e98f410ae39777f P da207f36066242e5bd3fcf2eec360572: Time spent starting tablet: real 0.006s user 0.003s sys 0.003s
I20260927 14:51:49.539219 4981 raft_consensus.cc:482] T 500815feacff4dc18e98f410ae39777f P d63f82fe261947e5946447215824a375 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 14:51:49.539475 4981 raft_consensus.cc:504] T 500815feacff4dc18e98f410ae39777f P d63f82fe261947e5946447215824a375 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d63f82fe261947e5946447215824a375" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35775 } } peers { permanent_uuid: "da207f36066242e5bd3fcf2eec360572" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39317 } } peers { permanent_uuid: "3b962746e285451ca0add90ca3ca2dd4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43821 } }
I20260927 14:51:49.540625 4981 leader_election.cc:288] T 500815feacff4dc18e98f410ae39777f P d63f82fe261947e5946447215824a375 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers da207f36066242e5bd3fcf2eec360572 (127.0.98.1:39317), 3b962746e285451ca0add90ca3ca2dd4 (127.0.98.2:43821)
I20260927 14:51:49.544703 4971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:51:49.540998 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:46247 (local address 127.0.98.1:39317)
0927 14:51:49.541139 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0927 14:51:49.541144 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 14:51:49.541161 (+ 17us) server_negotiation.cc:407] Connection header received
0927 14:51:49.541340 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:51:49.541344 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:51:49.541416 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:51:49.541497 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 14:51:49.541827 (+ 330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:51:49.542447 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:51:49.543651 (+ 1204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:51:49.544108 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:51:49.544222 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:51:49.544272 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0927 14:51:49.544496 (+ 224us) server_negotiation.cc:299] Negotiation successful
0927 14:51:49.544568 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260927 14:51:49.544699 4959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:51:49.540871 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:39317 (local address 127.0.98.3:46247)
0927 14:51:49.541101 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0927 14:51:49.541117 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 14:51:49.541205 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:51:49.541557 (+ 352us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:51:49.541560 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:51:49.541578 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 14:51:49.541699 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:51:49.541712 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:51:49.542564 (+ 852us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:51:49.542567 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:51:49.543513 (+ 946us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:51:49.543522 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:51:49.544245 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:51:49.544249 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:51:49.544359 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:51:49.544390 (+ 31us) client_negotiation.cc:769] Sending connection context
0927 14:51:49.544462 (+ 72us) client_negotiation.cc:240] Negotiation successful
0927 14:51:49.544543 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122,"mutex_wait_us":43}
I20260927 14:51:49.544857 4973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:51:49.541005 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:44175 (local address 127.0.98.2:43821)
0927 14:51:49.541139 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0927 14:51:49.541144 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 14:51:49.541483 (+ 339us) server_negotiation.cc:407] Connection header received
0927 14:51:49.541622 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:51:49.541625 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:51:49.541677 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:51:49.541716 (+ 39us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 14:51:49.542124 (+ 408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:51:49.542686 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:51:49.543812 (+ 1126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:51:49.544284 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:51:49.544398 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:51:49.544473 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0927 14:51:49.544658 (+ 185us) server_negotiation.cc:299] Negotiation successful
0927 14:51:49.544737 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260927 14:51:49.545078 4983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:51:49.540898 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:43821 (local address 127.0.98.3:44175)
0927 14:51:49.541358 (+ 460us) negotiation.cc:107] Waiting for socket to connect
0927 14:51:49.541372 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 14:51:49.541494 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:51:49.541874 (+ 380us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:51:49.541877 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:51:49.541895 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 14:51:49.541993 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:51:49.541998 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:51:49.542803 (+ 805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:51:49.542806 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:51:49.543684 (+ 878us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:51:49.543700 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:51:49.544406 (+ 706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:51:49.544409 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:51:49.544527 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:51:49.544549 (+ 22us) client_negotiation.cc:769] Sending connection context
0927 14:51:49.544624 (+ 75us) client_negotiation.cc:240] Negotiation successful
0927 14:51:49.544687 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":399,"mutex_wait_us":1,"thread_start_us":148,"threads_started":1}
I20260927 14:51:49.545503 4645 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "500815feacff4dc18e98f410ae39777f" candidate_uuid: "d63f82fe261947e5946447215824a375" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "da207f36066242e5bd3fcf2eec360572" is_pre_election: true
I20260927 14:51:49.545647 4778 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "500815feacff4dc18e98f410ae39777f" candidate_uuid: "d63f82fe261947e5946447215824a375" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b962746e285451ca0add90ca3ca2dd4" is_pre_election: true
I20260927 14:51:49.545842 4645 raft_consensus.cc:2454] T 500815feacff4dc18e98f410ae39777f P da207f36066242e5bd3fcf2eec360572 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d63f82fe261947e5946447215824a375 in term 0.
I20260927 14:51:49.545966 4778 raft_consensus.cc:2454] T 500815feacff4dc18e98f410ae39777f P 3b962746e285451ca0add90ca3ca2dd4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d63f82fe261947e5946447215824a375 in term 0.
I20260927 14:51:49.546392 4847 leader_election.cc:302] T 500815feacff4dc18e98f410ae39777f P d63f82fe261947e5946447215824a375 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 3b962746e285451ca0add90ca3ca2dd4, d63f82fe261947e5946447215824a375, da207f36066242e5bd3fcf2eec360572; no voters:
I20260927 14:51:49.546690 4981 raft_consensus.cc:2781] T 500815feacff4dc18e98f410ae39777f P d63f82fe261947e5946447215824a375 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 14:51:49.546789 4981 raft_consensus.cc:482] T 500815feacff4dc18e98f410ae39777f P d63f82fe261947e5946447215824a375 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 14:51:49.546862 4981 raft_consensus.cc:3037] T 500815feacff4dc18e98f410ae39777f P d63f82fe261947e5946447215824a375 [term 0 FOLLOWER]: Advancing to term 1
I20260927 14:51:49.547874 4981 raft_consensus.cc:504] T 500815feacff4dc18e98f410ae39777f P d63f82fe261947e5946447215824a375 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d63f82fe261947e5946447215824a375" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35775 } } peers { permanent_uuid: "da207f36066242e5bd3fcf2eec360572" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39317 } } peers { permanent_uuid: "3b962746e285451ca0add90ca3ca2dd4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43821 } }
I20260927 14:51:49.548244 4981 leader_election.cc:288] T 500815feacff4dc18e98f410ae39777f P d63f82fe261947e5946447215824a375 [CANDIDATE]: Term 1 election: Requested vote from peers da207f36066242e5bd3fcf2eec360572 (127.0.98.1:39317), 3b962746e285451ca0add90ca3ca2dd4 (127.0.98.2:43821)
I20260927 14:51:49.548604 4778 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "500815feacff4dc18e98f410ae39777f" candidate_uuid: "d63f82fe261947e5946447215824a375" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b962746e285451ca0add90ca3ca2dd4"
I20260927 14:51:49.548682 4645 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "500815feacff4dc18e98f410ae39777f" candidate_uuid: "d63f82fe261947e5946447215824a375" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "da207f36066242e5bd3fcf2eec360572"
I20260927 14:51:49.548725 4778 raft_consensus.cc:3037] T 500815feacff4dc18e98f410ae39777f P 3b962746e285451ca0add90ca3ca2dd4 [term 0 FOLLOWER]: Advancing to term 1
I20260927 14:51:49.548889 4645 raft_consensus.cc:3037] T 500815feacff4dc18e98f410ae39777f P da207f36066242e5bd3fcf2eec360572 [term 0 FOLLOWER]: Advancing to term 1
I20260927 14:51:49.549840 4778 raft_consensus.cc:2454] T 500815feacff4dc18e98f410ae39777f P 3b962746e285451ca0add90ca3ca2dd4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d63f82fe261947e5946447215824a375 in term 1.
I20260927 14:51:49.549957 4645 raft_consensus.cc:2454] T 500815feacff4dc18e98f410ae39777f P da207f36066242e5bd3fcf2eec360572 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d63f82fe261947e5946447215824a375 in term 1.
I20260927 14:51:49.550197 4846 leader_election.cc:302] T 500815feacff4dc18e98f410ae39777f P d63f82fe261947e5946447215824a375 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b962746e285451ca0add90ca3ca2dd4, d63f82fe261947e5946447215824a375; no voters:
I20260927 14:51:49.550398 4981 raft_consensus.cc:2781] T 500815feacff4dc18e98f410ae39777f P d63f82fe261947e5946447215824a375 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 14:51:49.550635 4981 raft_consensus.cc:686] T 500815feacff4dc18e98f410ae39777f P d63f82fe261947e5946447215824a375 [term 1 LEADER]: Becoming Leader. State: Replica: d63f82fe261947e5946447215824a375, State: Running, Role: LEADER
I20260927 14:51:49.550976 4981 consensus_queue.cc:237] T 500815feacff4dc18e98f410ae39777f P d63f82fe261947e5946447215824a375 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d63f82fe261947e5946447215824a375" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35775 } } peers { permanent_uuid: "da207f36066242e5bd3fcf2eec360572" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39317 } } peers { permanent_uuid: "3b962746e285451ca0add90ca3ca2dd4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43821 } }
I20260927 14:51:49.554147 4500 catalog_manager.cc:5495] T 500815feacff4dc18e98f410ae39777f P d63f82fe261947e5946447215824a375 reported cstate change: term changed from 0 to 1, leader changed from <none> to d63f82fe261947e5946447215824a375 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "d63f82fe261947e5946447215824a375" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d63f82fe261947e5946447215824a375" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35775 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "da207f36066242e5bd3fcf2eec360572" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39317 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3b962746e285451ca0add90ca3ca2dd4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43821 } health_report { overall_health: UNKNOWN } } }
I20260927 14:51:49.567171 4972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:51:49.563574 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55312 (local address 127.0.98.3:35775)
0927 14:51:49.563699 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0927 14:51:49.563702 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 14:51:49.563716 (+ 14us) server_negotiation.cc:407] Connection header received
0927 14:51:49.563851 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:51:49.563853 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:51:49.563905 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:51:49.563991 (+ 86us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 14:51:49.564251 (+ 260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:51:49.564735 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:51:49.565653 (+ 918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:51:49.565877 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:51:49.565990 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:51:49.566316 (+ 326us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 14:51:49.566749 (+ 433us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 14:51:49.566887 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0927 14:51:49.566950 (+ 63us) server_negotiation.cc:299] Negotiation successful
0927 14:51:49.567007 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260927 14:51:49.572897 4645 raft_consensus.cc:1261] T 500815feacff4dc18e98f410ae39777f P da207f36066242e5bd3fcf2eec360572 [term 1 FOLLOWER]: Refusing update from remote peer d63f82fe261947e5946447215824a375: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 14:51:49.572907 4778 raft_consensus.cc:1261] T 500815feacff4dc18e98f410ae39777f P 3b962746e285451ca0add90ca3ca2dd4 [term 1 FOLLOWER]: Refusing update from remote peer d63f82fe261947e5946447215824a375: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 14:51:49.573627 4981 consensus_queue.cc:1035] T 500815feacff4dc18e98f410ae39777f P d63f82fe261947e5946447215824a375 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b962746e285451ca0add90ca3ca2dd4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43821 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 14:51:49.573877 4984 consensus_queue.cc:1035] T 500815feacff4dc18e98f410ae39777f P d63f82fe261947e5946447215824a375 [LEADER]: Connected to new peer: Peer: permanent_uuid: "da207f36066242e5bd3fcf2eec360572" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39317 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 14:51:49.582829 4989 mvcc.cc:205] Tried to move back new op lower bound from 7333972826403028992 to 7333972826323255296. Current Snapshot: MvccSnapshot[applied={T|T < 7333972826403028992}]
I20260927 14:51:49.591138 4499 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52604:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 14:51:49.591351 4499 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52604:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 14:51:49.593822 4499 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 713317bf382b4c8289e4b49012cb3d2d: Sending DeleteTablet for 3 replicas of tablet 500815feacff4dc18e98f410ae39777f
I20260927 14:51:49.594726 4625 tablet_service.cc:1514] Processing DeleteTablet for tablet 500815feacff4dc18e98f410ae39777f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 14:51:49 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:49258
I20260927 14:51:49.594726 4758 tablet_service.cc:1514] Processing DeleteTablet for tablet 500815feacff4dc18e98f410ae39777f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 14:51:49 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:55106
I20260927 14:51:49.595306 4998 tablet_replica.cc:330] T 500815feacff4dc18e98f410ae39777f P 3b962746e285451ca0add90ca3ca2dd4: stopping tablet replica
I20260927 14:51:49.595428 4891 tablet_service.cc:1514] Processing DeleteTablet for tablet 500815feacff4dc18e98f410ae39777f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 14:51:49 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:55302
I20260927 14:51:49.595715 4998 raft_consensus.cc:2229] T 500815feacff4dc18e98f410ae39777f P 3b962746e285451ca0add90ca3ca2dd4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 14:51:49.595818 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6936m5/build/debug/bin/kudu with pid 4563
I20260927 14:51:49.595965 4999 tablet_replica.cc:330] T 500815feacff4dc18e98f410ae39777f P d63f82fe261947e5946447215824a375: stopping tablet replica
I20260927 14:51:49.596043 4998 raft_consensus.cc:2258] T 500815feacff4dc18e98f410ae39777f P 3b962746e285451ca0add90ca3ca2dd4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 14:51:49.597572 4998 ts_tablet_manager.cc:1905] T 500815feacff4dc18e98f410ae39777f P 3b962746e285451ca0add90ca3ca2dd4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 14:51:49.597575 4999 raft_consensus.cc:2229] T 500815feacff4dc18e98f410ae39777f P d63f82fe261947e5946447215824a375 [term 1 LEADER]: Raft consensus shutting down.
I20260927 14:51:49.598284 4999 raft_consensus.cc:2258] T 500815feacff4dc18e98f410ae39777f P d63f82fe261947e5946447215824a375 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 14:51:49.599800 4999 ts_tablet_manager.cc:1905] T 500815feacff4dc18e98f410ae39777f P d63f82fe261947e5946447215824a375: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 14:51:49.600344 4998 ts_tablet_manager.cc:1918] T 500815feacff4dc18e98f410ae39777f P 3b962746e285451ca0add90ca3ca2dd4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 14:51:49.600492 4998 log.cc:1196] T 500815feacff4dc18e98f410ae39777f P 3b962746e285451ca0add90ca3ca2dd4: Deleting WAL directory at /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790520684602251-392-0/minicluster-data/ts-1/wal/wals/500815feacff4dc18e98f410ae39777f
I20260927 14:51:49.600874 4998 ts_tablet_manager.cc:1939] T 500815feacff4dc18e98f410ae39777f P 3b962746e285451ca0add90ca3ca2dd4: Deleting consensus metadata
I20260927 14:51:49.602157 4487 catalog_manager.cc:4841] TS 3b962746e285451ca0add90ca3ca2dd4 (127.0.98.2:43821): tablet 500815feacff4dc18e98f410ae39777f (table test-table [id=e7638b4dd64f438cbbd01a4ce0f1ffb6]) successfully deleted
I20260927 14:51:49.603329 4999 ts_tablet_manager.cc:1918] T 500815feacff4dc18e98f410ae39777f P d63f82fe261947e5946447215824a375: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 14:51:49.603436 4999 log.cc:1196] T 500815feacff4dc18e98f410ae39777f P d63f82fe261947e5946447215824a375: Deleting WAL directory at /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790520684602251-392-0/minicluster-data/ts-2/wal/wals/500815feacff4dc18e98f410ae39777f
I20260927 14:51:49.603833 4999 ts_tablet_manager.cc:1939] T 500815feacff4dc18e98f410ae39777f P d63f82fe261947e5946447215824a375: Deleting consensus metadata
W20260927 14:51:49.604600 4488 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:39317 (error 108)
I20260927 14:51:49.604648 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6936m5/build/debug/bin/kudu with pid 4696
I20260927 14:51:49.605870 4970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:51:49.605569 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:39317 (local address 127.0.0.1:49274)
0927 14:51:49.605730 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0927 14:51:49.605789 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:39317: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":89}
W20260927 14:51:49.606104 4488 catalog_manager.cc:4584] TS da207f36066242e5bd3fcf2eec360572 (127.0.98.1:39317): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 500815feacff4dc18e98f410ae39777f: Network error: Client connection negotiation failed: client connection to 127.0.98.1:39317: connect: Connection refused (error 111)
I20260927 14:51:49.609885 4487 catalog_manager.cc:4841] TS d63f82fe261947e5946447215824a375 (127.0.98.3:35775): tablet 500815feacff4dc18e98f410ae39777f (table test-table [id=e7638b4dd64f438cbbd01a4ce0f1ffb6]) successfully deleted
I20260927 14:51:49.610471 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6936m5/build/debug/bin/kudu with pid 4829
I20260927 14:51:49.616199 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6936m5/build/debug/bin/kudu with pid 4470
2026-09-27T14:51:49Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (2323 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-09-27T14:51:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T14:51:49Z Disabled control of system clock
I20260927 14:51:49.645442 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6936m5/build/debug/bin/kudu
/tmp/dist-test-task6936m5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790520684602251-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790520684602251-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790520684602251-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790520684602251-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37483
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:36533
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:37483
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260927 14:51:49.753376 5004 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 14:51:49.753691 5004 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 14:51:49.753787 5004 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 14:51:49.758450 5004 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 14:51:49.758517 5004 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 14:51:49.758533 5004 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 14:51:49.758551 5004 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 14:51:49.758566 5004 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 14:51:49.763602 5004 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36533
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790520684602251-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790520684602251-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:37483
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37483
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790520684602251-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5004
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790520684602251-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:15:17 UTC on ccycloud.quasar-ledxfe.root.comops.site
build id 4676134
I20260927 14:51:49.764755 5004 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 14:51:49.766407 5004 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 14:51:49.775256 5012 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 14:51:49.775276 5010 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 14:51:49.775260 5009 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 14:51:49.775520 5004 server_base.cc:1034] running on GCE node
I20260927 14:51:49.776117 5004 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 14:51:49.777271 5004 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 14:51:49.778453 5004 hybrid_clock.cc:648] HybridClock initialized: now 1790520709778437 us; error 26 us; skew 500 ppm
I20260927 14:51:49.780294 5004 webserver.cc:460] Webserver started at http://127.0.98.62:39763/ using document root <none> and password file <none>
I20260927 14:51:49.780900 5004 fs_manager.cc:359] Metadata directory not provided
I20260927 14:51:49.780963 5004 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 14:51:49.781335 5004 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 14:51:49.783448 5004 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790520684602251-392-0/minicluster-data/master-0/data/instance:
uuid: "67a82d6810144c32be0112f256962174"
format_stamp: "Formatted at 2026-09-27 14:51:49 on dist-test-slave-xk1g"
server_key: "f0b1fde533e13a6c149e668661e198a2"
server_key_iv: "d4b0beeaf3cfcf2ffe7ebafda6ee30f3"
server_key_version: "encryptionkey@0"
I20260927 14:51:49.784062 5004 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790520684602251-392-0/minicluster-data/master-0/wal/instance:
uuid: "67a82d6810144c32be0112f256962174"
format_stamp: "Formatted at 2026-09-27 14:51:49 on dist-test-slave-xk1g"
server_key: "f0b1fde533e13a6c149e668661e198a2"
server_key_iv: "d4b0beeaf3cfcf2ffe7ebafda6ee30f3"
server_key_version: "encryptionkey@0"
I20260927 14:51:49.787874 5004 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.004s
I20260927 14:51:49.790537 5018 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 14:51:49.792096 5004 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260927 14:51:49.792250 5004 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790520684602251-392-0/minicluster-data/master-0/data,/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790520684602251-392-0/minicluster-data/master-0/wal
uuid: "67a82d6810144c32be0112f256962174"
format_stamp: "Formatted at 2026-09-27 14:51:49 on dist-test-slave-xk1g"
server_key: "f0b1fde533e13a6c149e668661e198a2"
server_key_iv: "d4b0beeaf3cfcf2ffe7ebafda6ee30f3"
server_key_version: "encryptionkey@0"
I20260927 14:51:49.792411 5004 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790520684602251-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790520684602251-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790520684602251-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 14:51:49.798202 5004 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 14:51:49.798756 5004 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 14:51:49.798905 5004 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 14:51:49.805778 5004 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:37483
I20260927 14:51:49.805833 5070 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:37483 every 8 connection(s)
I20260927 14:51:49.806777 5004 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790520684602251-392-0/minicluster-data/master-0/data/info.pb
I20260927 14:51:49.810621 5071 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 14:51:49.810665 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6936m5/build/debug/bin/kudu as pid 5004
I20260927 14:51:49.810786 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790520684602251-392-0/minicluster-data/master-0/wal/instance
I20260927 14:51:49.811164 392 external_mini_cluster.cc:1428] Setting key da9bd7cf19cb10463eb44cac4bcbb288
I20260927 14:51:49.817202 5074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:51:49.812830 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44092 (local address 127.0.98.62:37483)
0927 14:51:49.813538 (+ 708us) server_negotiation.cc:206] Beginning negotiation
0927 14:51:49.813554 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0927 14:51:49.813590 (+ 36us) server_negotiation.cc:407] Connection header received
0927 14:51:49.814298 (+ 708us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:51:49.814315 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:51:49.814668 (+ 353us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:51:49.815017 (+ 349us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:51:49.815864 (+ 847us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:51:49.815870 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:51:49.815882 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 14:51:49.815892 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:51:49.815949 (+ 57us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 14:51:49.815967 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:51:49.816128 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0927 14:51:49.816203 (+ 75us) server_negotiation.cc:299] Negotiation successful
0927 14:51:49.816444 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":346,"thread_start_us":181,"threads_started":1}
I20260927 14:51:49.818361 5071 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 67a82d6810144c32be0112f256962174: Bootstrap starting.
I20260927 14:51:49.820811 5071 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 67a82d6810144c32be0112f256962174: Neither blocks nor log segments found. Creating new log.
I20260927 14:51:49.821763 5071 log.cc:824] T 00000000000000000000000000000000 P 67a82d6810144c32be0112f256962174: Log is configured to *not* fsync() on all Append() calls
I20260927 14:51:49.824013 5071 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 67a82d6810144c32be0112f256962174: No bootstrap required, opened a new log
I20260927 14:51:49.828003 5071 raft_consensus.cc:348] T 00000000000000000000000000000000 P 67a82d6810144c32be0112f256962174 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "67a82d6810144c32be0112f256962174" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37483 } }
I20260927 14:51:49.828248 5071 raft_consensus.cc:374] T 00000000000000000000000000000000 P 67a82d6810144c32be0112f256962174 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 14:51:49.828320 5071 raft_consensus.cc:729] T 00000000000000000000000000000000 P 67a82d6810144c32be0112f256962174 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 67a82d6810144c32be0112f256962174, State: Initialized, Role: FOLLOWER
I20260927 14:51:49.828770 5071 consensus_queue.cc:260] T 00000000000000000000000000000000 P 67a82d6810144c32be0112f256962174 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "67a82d6810144c32be0112f256962174" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37483 } }
I20260927 14:51:49.828924 5071 raft_consensus.cc:388] T 00000000000000000000000000000000 P 67a82d6810144c32be0112f256962174 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 14:51:49.828995 5071 raft_consensus.cc:482] T 00000000000000000000000000000000 P 67a82d6810144c32be0112f256962174 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 14:51:49.829113 5071 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 67a82d6810144c32be0112f256962174 [term 0 FOLLOWER]: Advancing to term 1
I20260927 14:51:49.852248 5071 raft_consensus.cc:504] T 00000000000000000000000000000000 P 67a82d6810144c32be0112f256962174 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "67a82d6810144c32be0112f256962174" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37483 } }
I20260927 14:51:49.852739 5071 leader_election.cc:302] T 00000000000000000000000000000000 P 67a82d6810144c32be0112f256962174 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 67a82d6810144c32be0112f256962174; no voters:
I20260927 14:51:49.853174 5071 leader_election.cc:288] T 00000000000000000000000000000000 P 67a82d6810144c32be0112f256962174 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 14:51:49.853271 5076 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 67a82d6810144c32be0112f256962174 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 14:51:49.853536 5076 raft_consensus.cc:686] T 00000000000000000000000000000000 P 67a82d6810144c32be0112f256962174 [term 1 LEADER]: Becoming Leader. State: Replica: 67a82d6810144c32be0112f256962174, State: Running, Role: LEADER
I20260927 14:51:49.853945 5076 consensus_queue.cc:237] T 00000000000000000000000000000000 P 67a82d6810144c32be0112f256962174 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "67a82d6810144c32be0112f256962174" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37483 } }
I20260927 14:51:49.854691 5071 sys_catalog.cc:564] T 00000000000000000000000000000000 P 67a82d6810144c32be0112f256962174 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 14:51:49.855866 5077 sys_catalog.cc:455] T 00000000000000000000000000000000 P 67a82d6810144c32be0112f256962174 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "67a82d6810144c32be0112f256962174" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "67a82d6810144c32be0112f256962174" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37483 } } }
I20260927 14:51:49.855870 5078 sys_catalog.cc:455] T 00000000000000000000000000000000 P 67a82d6810144c32be0112f256962174 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 67a82d6810144c32be0112f256962174. Latest consensus state: current_term: 1 leader_uuid: "67a82d6810144c32be0112f256962174" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "67a82d6810144c32be0112f256962174" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37483 } } }
I20260927 14:51:49.856009 5077 sys_catalog.cc:458] T 00000000000000000000000000000000 P 67a82d6810144c32be0112f256962174 [sys.catalog]: This master's current role is: LEADER
I20260927 14:51:49.856020 5078 sys_catalog.cc:458] T 00000000000000000000000000000000 P 67a82d6810144c32be0112f256962174 [sys.catalog]: This master's current role is: LEADER
I20260927 14:51:49.856428 5085 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 14:51:49.859690 5085 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 14:51:49.865139 5085 catalog_manager.cc:1347] Generated new cluster ID: 48e299c3ba7c4669b28f0b97e03aadc0
I20260927 14:51:49.865211 5085 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 14:51:49.897789 5085 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 14:51:49.898263 5085 catalog_manager.cc:1504] Loading token signing keys...
I20260927 14:51:49.909003 5085 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 67a82d6810144c32be0112f256962174: Generated new TSK 0
I20260927 14:51:49.909798 5085 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 14:51:49.916018 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6936m5/build/debug/bin/kudu
/tmp/dist-test-task6936m5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790520684602251-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790520684602251-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790520684602251-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790520684602251-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37483
--builtin_ntp_servers=127.0.98.20:36533
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260927 14:51:50.045370 5095 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 14:51:50.045615 5095 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 14:51:50.045683 5095 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 14:51:50.050315 5095 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 14:51:50.050364 5095 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 14:51:50.050419 5095 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 14:51:50.056090 5095 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36533
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790520684602251-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790520684602251-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790520684602251-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37483
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5095
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790520684602251-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:15:17 UTC on ccycloud.quasar-ledxfe.root.comops.site
build id 4676134
I20260927 14:51:50.057245 5095 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 14:51:50.058645 5095 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 14:51:50.068732 5103 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 14:51:50.068740 5100 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 14:51:50.068740 5101 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 14:51:50.068990 5095 server_base.cc:1034] running on GCE node
I20260927 14:51:50.069486 5095 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 14:51:50.070066 5095 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 14:51:50.071213 5095 hybrid_clock.cc:648] HybridClock initialized: now 1790520710071181 us; error 44 us; skew 500 ppm
I20260927 14:51:50.072989 5095 webserver.cc:460] Webserver started at http://127.0.98.1:37749/ using document root <none> and password file <none>
I20260927 14:51:50.073614 5095 fs_manager.cc:359] Metadata directory not provided
I20260927 14:51:50.073675 5095 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 14:51:50.073834 5095 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 14:51:50.076058 5095 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790520684602251-392-0/minicluster-data/ts-0/data/instance:
uuid: "aa9a06d24d58498bafdc611d5777e5cf"
format_stamp: "Formatted at 2026-09-27 14:51:50 on dist-test-slave-xk1g"
server_key: "deecff9b4550a3c3896b38d6d5bdfb4a"
server_key_iv: "23189909d3bb6b9701b9ec38eb3cdaab"
server_key_version: "encryptionkey@0"
I20260927 14:51:50.076581 5095 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790520684602251-392-0/minicluster-data/ts-0/wal/instance:
uuid: "aa9a06d24d58498bafdc611d5777e5cf"
format_stamp: "Formatted at 2026-09-27 14:51:50 on dist-test-slave-xk1g"
server_key: "deecff9b4550a3c3896b38d6d5bdfb4a"
server_key_iv: "23189909d3bb6b9701b9ec38eb3cdaab"
server_key_version: "encryptionkey@0"
I20260927 14:51:50.080443 5095 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.003s
I20260927 14:51:50.083024 5109 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 14:51:50.084275 5095 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260927 14:51:50.084394 5095 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790520684602251-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790520684602251-392-0/minicluster-data/ts-0/wal
uuid: "aa9a06d24d58498bafdc611d5777e5cf"
format_stamp: "Formatted at 2026-09-27 14:51:50 on dist-test-slave-xk1g"
server_key: "deecff9b4550a3c3896b38d6d5bdfb4a"
server_key_iv: "23189909d3bb6b9701b9ec38eb3cdaab"
server_key_version: "encryptionkey@0"
I20260927 14:51:50.084501 5095 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790520684602251-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790520684602251-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790520684602251-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 14:51:50.089705 5095 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 14:51:50.090401 5095 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 14:51:50.090597 5095 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 14:51:50.091156 5095 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 14:51:50.092123 5095 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 14:51:50.092195 5095 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 14:51:50.092257 5095 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 14:51:50.092288 5095 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 14:51:50.101819 5095 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:37973
I20260927 14:51:50.101836 5222 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:37973 every 8 connection(s)
I20260927 14:51:50.102797 5095 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790520684602251-392-0/minicluster-data/ts-0/data/info.pb
I20260927 14:51:50.109133 5074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:51:50.104654 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:39175 (local address 127.0.98.62:37483)
0927 14:51:50.104877 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0927 14:51:50.104883 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 14:51:50.106014 (+ 1131us) server_negotiation.cc:407] Connection header received
0927 14:51:50.107291 (+ 1277us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:51:50.107294 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:51:50.107342 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:51:50.107429 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:51:50.108654 (+ 1225us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:51:50.108657 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:51:50.108659 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 14:51:50.108671 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:51:50.108689 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 14:51:50.108697 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:51:50.108782 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0927 14:51:50.108912 (+ 130us) server_negotiation.cc:299] Negotiation successful
0927 14:51:50.108948 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260927 14:51:50.110021 5225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:51:50.105174 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37483 (local address 127.0.98.1:39175)
0927 14:51:50.105844 (+ 670us) negotiation.cc:107] Waiting for socket to connect
0927 14:51:50.105885 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0927 14:51:50.107066 (+ 1181us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:51:50.107625 (+ 559us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:51:50.107632 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:51:50.107727 (+ 95us) client_negotiation.cc:189] Negotiated authn=SASL
0927 14:51:50.108384 (+ 657us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 14:51:50.108386 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0927 14:51:50.108435 (+ 49us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 14:51:50.108509 (+ 74us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 14:51:50.108783 (+ 274us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 14:51:50.108790 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 14:51:50.108794 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 14:51:50.108941 (+ 147us) client_negotiation.cc:240] Negotiation successful
0927 14:51:50.109220 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":271,"thread_start_us":135,"threads_started":1}
I20260927 14:51:50.111533 5223 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37483
I20260927 14:51:50.111807 5223 heartbeater.cc:464] Registering TS with master...
I20260927 14:51:50.112273 5223 heartbeater.cc:511] Master 127.0.98.62:37483 requested a full tablet report, sending...
I20260927 14:51:50.113018 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6936m5/build/debug/bin/kudu as pid 5095
I20260927 14:51:50.113227 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790520684602251-392-0/minicluster-data/ts-0/wal/instance
I20260927 14:51:50.113504 392 external_mini_cluster.cc:1428] Setting key f4c6d5b16f7a89e9a34112fcff97d160
I20260927 14:51:50.114085 5035 ts_manager.cc:194] Registered new tserver with Master: aa9a06d24d58498bafdc611d5777e5cf (127.0.98.1:37973)
I20260927 14:51:50.115728 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6936m5/build/debug/bin/kudu
/tmp/dist-test-task6936m5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790520684602251-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790520684602251-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790520684602251-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790520684602251-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37483
--builtin_ntp_servers=127.0.98.20:36533
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260927 14:51:50.227751 5226 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 14:51:50.228049 5226 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 14:51:50.228117 5226 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 14:51:50.232097 5226 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 14:51:50.232180 5226 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 14:51:50.232277 5226 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 14:51:50.237105 5226 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36533
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790520684602251-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790520684602251-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790520684602251-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37483
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5226
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790520684602251-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:15:17 UTC on ccycloud.quasar-ledxfe.root.comops.site
build id 4676134
I20260927 14:51:50.238234 5226 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 14:51:50.239601 5226 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 14:51:50.249025 5232 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 14:51:50.249012 5234 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 14:51:50.249286 5226 server_base.cc:1034] running on GCE node
W20260927 14:51:50.249419 5231 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 14:51:50.249792 5226 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 14:51:50.250442 5226 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 14:51:50.251624 5226 hybrid_clock.cc:648] HybridClock initialized: now 1790520710251595 us; error 42 us; skew 500 ppm
I20260927 14:51:50.253561 5226 webserver.cc:460] Webserver started at http://127.0.98.2:42019/ using document root <none> and password file <none>
I20260927 14:51:50.254142 5226 fs_manager.cc:359] Metadata directory not provided
I20260927 14:51:50.254241 5226 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 14:51:50.254470 5226 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 14:51:50.256954 5226 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790520684602251-392-0/minicluster-data/ts-1/data/instance:
uuid: "2ee1ebb2fee64a9a98469148f2d7e0a2"
format_stamp: "Formatted at 2026-09-27 14:51:50 on dist-test-slave-xk1g"
server_key: "3b31d92bbd8a64d96838cffcb3a79b90"
server_key_iv: "6135b4644c762308e2eea05d5de795db"
server_key_version: "encryptionkey@0"
I20260927 14:51:50.257594 5226 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790520684602251-392-0/minicluster-data/ts-1/wal/instance:
uuid: "2ee1ebb2fee64a9a98469148f2d7e0a2"
format_stamp: "Formatted at 2026-09-27 14:51:50 on dist-test-slave-xk1g"
server_key: "3b31d92bbd8a64d96838cffcb3a79b90"
server_key_iv: "6135b4644c762308e2eea05d5de795db"
server_key_version: "encryptionkey@0"
I20260927 14:51:50.261482 5226 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260927 14:51:50.263861 5240 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 14:51:50.265174 5226 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 14:51:50.265300 5226 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790520684602251-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790520684602251-392-0/minicluster-data/ts-1/wal
uuid: "2ee1ebb2fee64a9a98469148f2d7e0a2"
format_stamp: "Formatted at 2026-09-27 14:51:50 on dist-test-slave-xk1g"
server_key: "3b31d92bbd8a64d96838cffcb3a79b90"
server_key_iv: "6135b4644c762308e2eea05d5de795db"
server_key_version: "encryptionkey@0"
I20260927 14:51:50.265419 5226 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790520684602251-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790520684602251-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790520684602251-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 14:51:50.269836 5226 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 14:51:50.270412 5226 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 14:51:50.270582 5226 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 14:51:50.271088 5226 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 14:51:50.271927 5226 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 14:51:50.271996 5226 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 14:51:50.272060 5226 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 14:51:50.272112 5226 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 14:51:50.281625 5226 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:46497
I20260927 14:51:50.281670 5353 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:46497 every 8 connection(s)
I20260927 14:51:50.282469 5226 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790520684602251-392-0/minicluster-data/ts-1/data/info.pb
I20260927 14:51:50.288573 5074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:51:50.284166 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:54435 (local address 127.0.98.62:37483)
0927 14:51:50.284311 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0927 14:51:50.284315 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 14:51:50.285306 (+ 991us) server_negotiation.cc:407] Connection header received
0927 14:51:50.286597 (+ 1291us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:51:50.286601 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:51:50.286650 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:51:50.286740 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:51:50.288069 (+ 1329us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:51:50.288073 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:51:50.288075 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 14:51:50.288090 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:51:50.288113 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 14:51:50.288130 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:51:50.288232 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0927 14:51:50.288351 (+ 119us) server_negotiation.cc:299] Negotiation successful
0927 14:51:50.288415 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260927 14:51:50.289500 5356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:51:50.284473 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37483 (local address 127.0.98.2:54435)
0927 14:51:50.285153 (+ 680us) negotiation.cc:107] Waiting for socket to connect
0927 14:51:50.285192 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0927 14:51:50.286383 (+ 1191us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:51:50.286896 (+ 513us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:51:50.286923 (+ 27us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:51:50.287034 (+ 111us) client_negotiation.cc:189] Negotiated authn=SASL
0927 14:51:50.287721 (+ 687us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 14:51:50.287726 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 14:51:50.287779 (+ 53us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 14:51:50.287919 (+ 140us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 14:51:50.288216 (+ 297us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 14:51:50.288223 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 14:51:50.288229 (+ 6us) client_negotiation.cc:769] Sending connection context
0927 14:51:50.288409 (+ 180us) client_negotiation.cc:240] Negotiation successful
0927 14:51:50.288637 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":236,"thread_start_us":101,"threads_started":1}
I20260927 14:51:50.290560 5354 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37483
I20260927 14:51:50.290750 5354 heartbeater.cc:464] Registering TS with master...
I20260927 14:51:50.291126 5354 heartbeater.cc:511] Master 127.0.98.62:37483 requested a full tablet report, sending...
I20260927 14:51:50.291947 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6936m5/build/debug/bin/kudu as pid 5226
I20260927 14:51:50.292071 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790520684602251-392-0/minicluster-data/ts-1/wal/instance
I20260927 14:51:50.292083 5035 ts_manager.cc:194] Registered new tserver with Master: 2ee1ebb2fee64a9a98469148f2d7e0a2 (127.0.98.2:46497)
I20260927 14:51:50.292357 392 external_mini_cluster.cc:1428] Setting key 111bf30197a04ef34212e5d6998db1ba
I20260927 14:51:50.294322 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6936m5/build/debug/bin/kudu
/tmp/dist-test-task6936m5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790520684602251-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790520684602251-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790520684602251-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790520684602251-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37483
--builtin_ntp_servers=127.0.98.20:36533
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260927 14:51:50.404031 5357 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 14:51:50.404275 5357 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 14:51:50.404321 5357 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 14:51:50.408332 5357 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 14:51:50.408421 5357 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 14:51:50.408511 5357 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 14:51:50.413192 5357 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36533
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790520684602251-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790520684602251-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790520684602251-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37483
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5357
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790520684602251-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:15:17 UTC on ccycloud.quasar-ledxfe.root.comops.site
build id 4676134
I20260927 14:51:50.414197 5357 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 14:51:50.415498 5357 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 14:51:50.424953 5365 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 14:51:50.424955 5362 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 14:51:50.425001 5357 server_base.cc:1034] running on GCE node
W20260927 14:51:50.424955 5363 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 14:51:50.425689 5357 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 14:51:50.426301 5357 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 14:51:50.427466 5357 hybrid_clock.cc:648] HybridClock initialized: now 1790520710427416 us; error 65 us; skew 500 ppm
I20260927 14:51:50.429535 5357 webserver.cc:460] Webserver started at http://127.0.98.3:34847/ using document root <none> and password file <none>
I20260927 14:51:50.430137 5357 fs_manager.cc:359] Metadata directory not provided
I20260927 14:51:50.430198 5357 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 14:51:50.430426 5357 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 14:51:50.432488 5357 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790520684602251-392-0/minicluster-data/ts-2/data/instance:
uuid: "b93265bf78594e7392160e7bbe2a6df8"
format_stamp: "Formatted at 2026-09-27 14:51:50 on dist-test-slave-xk1g"
server_key: "8051388d35d5f827ef734d2a57cf4872"
server_key_iv: "e3b4a989566f753dcc6f83dfe8ef4496"
server_key_version: "encryptionkey@0"
I20260927 14:51:50.433045 5357 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790520684602251-392-0/minicluster-data/ts-2/wal/instance:
uuid: "b93265bf78594e7392160e7bbe2a6df8"
format_stamp: "Formatted at 2026-09-27 14:51:50 on dist-test-slave-xk1g"
server_key: "8051388d35d5f827ef734d2a57cf4872"
server_key_iv: "e3b4a989566f753dcc6f83dfe8ef4496"
server_key_version: "encryptionkey@0"
I20260927 14:51:50.436707 5357 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260927 14:51:50.439254 5371 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 14:51:50.440546 5357 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 14:51:50.440678 5357 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790520684602251-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790520684602251-392-0/minicluster-data/ts-2/wal
uuid: "b93265bf78594e7392160e7bbe2a6df8"
format_stamp: "Formatted at 2026-09-27 14:51:50 on dist-test-slave-xk1g"
server_key: "8051388d35d5f827ef734d2a57cf4872"
server_key_iv: "e3b4a989566f753dcc6f83dfe8ef4496"
server_key_version: "encryptionkey@0"
I20260927 14:51:50.440788 5357 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790520684602251-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790520684602251-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790520684602251-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 14:51:50.445276 5357 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 14:51:50.445838 5357 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 14:51:50.445977 5357 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 14:51:50.446430 5357 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 14:51:50.447225 5357 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 14:51:50.447270 5357 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 14:51:50.447305 5357 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 14:51:50.447320 5357 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 14:51:50.455745 5357 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:40405
I20260927 14:51:50.455812 5484 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:40405 every 8 connection(s)
I20260927 14:51:50.456674 5357 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790520684602251-392-0/minicluster-data/ts-2/data/info.pb
I20260927 14:51:50.460211 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6936m5/build/debug/bin/kudu as pid 5357
I20260927 14:51:50.460383 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790520684602251-392-0/minicluster-data/ts-2/wal/instance
I20260927 14:51:50.460657 392 external_mini_cluster.cc:1428] Setting key aa7b12a71fffd20dc55967007de56258
I20260927 14:51:50.462817 5074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:51:50.458425 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:42887 (local address 127.0.98.62:37483)
0927 14:51:50.458567 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0927 14:51:50.458572 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 14:51:50.459679 (+ 1107us) server_negotiation.cc:407] Connection header received
0927 14:51:50.461112 (+ 1433us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:51:50.461115 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:51:50.461160 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:51:50.461268 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:51:50.462366 (+ 1098us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:51:50.462370 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:51:50.462372 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 14:51:50.462385 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:51:50.462410 (+ 25us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 14:51:50.462418 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:51:50.462510 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0927 14:51:50.462647 (+ 137us) server_negotiation.cc:299] Negotiation successful
0927 14:51:50.462691 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260927 14:51:50.463661 5487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:51:50.458858 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37483 (local address 127.0.98.3:42887)
0927 14:51:50.459503 (+ 645us) negotiation.cc:107] Waiting for socket to connect
0927 14:51:50.459540 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0927 14:51:50.460879 (+ 1339us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:51:50.461429 (+ 550us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:51:50.461443 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:51:50.461530 (+ 87us) client_negotiation.cc:189] Negotiated authn=SASL
0927 14:51:50.462120 (+ 590us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 14:51:50.462122 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0927 14:51:50.462159 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 14:51:50.462224 (+ 65us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 14:51:50.462500 (+ 276us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 14:51:50.462507 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 14:51:50.462513 (+ 6us) client_negotiation.cc:769] Sending connection context
0927 14:51:50.462691 (+ 178us) client_negotiation.cc:240] Negotiation successful
0927 14:51:50.462917 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":248,"thread_start_us":111,"threads_started":1}
I20260927 14:51:50.464659 5485 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37483
I20260927 14:51:50.464854 5485 heartbeater.cc:464] Registering TS with master...
I20260927 14:51:50.465245 5485 heartbeater.cc:511] Master 127.0.98.62:37483 requested a full tablet report, sending...
I20260927 14:51:50.466202 5035 ts_manager.cc:194] Registered new tserver with Master: b93265bf78594e7392160e7bbe2a6df8 (127.0.98.3:40405)
I20260927 14:51:50.474650 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 14:51:50.478160 5074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:51:50.476624 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44106 (local address 127.0.98.62:37483)
0927 14:51:50.476751 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0927 14:51:50.476754 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 14:51:50.476970 (+ 216us) server_negotiation.cc:407] Connection header received
0927 14:51:50.477091 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:51:50.477093 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:51:50.477158 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:51:50.477222 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:51:50.477867 (+ 645us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:51:50.477870 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:51:50.477872 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 14:51:50.477882 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:51:50.477899 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 14:51:50.477906 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:51:50.477981 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0927 14:51:50.478032 (+ 51us) server_negotiation.cc:299] Negotiation successful
0927 14:51:50.478064 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260927 14:51:50.480865 5035 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44106:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 14:51:50.483919 5035 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 14:51:50.495579 5500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:51:50.492928 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:40405 (local address 127.0.0.1:45796)
0927 14:51:50.493552 (+ 624us) negotiation.cc:107] Waiting for socket to connect
0927 14:51:50.493565 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 14:51:50.493629 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:51:50.494075 (+ 446us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:51:50.494082 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:51:50.494157 (+ 75us) client_negotiation.cc:189] Negotiated authn=SASL
0927 14:51:50.494795 (+ 638us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 14:51:50.494799 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 14:51:50.494822 (+ 23us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 14:51:50.494898 (+ 76us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 14:51:50.495211 (+ 313us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 14:51:50.495220 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 14:51:50.495225 (+ 5us) client_negotiation.cc:769] Sending connection context
0927 14:51:50.495336 (+ 111us) client_negotiation.cc:240] Negotiation successful
0927 14:51:50.495419 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":563,"thread_start_us":81,"threads_started":1}
I20260927 14:51:50.495891 5498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:51:50.492272 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50554 (local address 127.0.98.2:46497)
0927 14:51:50.493073 (+ 801us) server_negotiation.cc:206] Beginning negotiation
0927 14:51:50.493091 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0927 14:51:50.493116 (+ 25us) server_negotiation.cc:407] Connection header received
0927 14:51:50.493168 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:51:50.493173 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:51:50.493372 (+ 199us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:51:50.493524 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:51:50.495043 (+ 1519us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:51:50.495053 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:51:50.495069 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 14:51:50.495082 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:51:50.495124 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 14:51:50.495158 (+ 34us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:51:50.495263 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0927 14:51:50.495644 (+ 381us) server_negotiation.cc:299] Negotiation successful
0927 14:51:50.495713 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":680,"thread_start_us":139,"threads_started":1}
I20260927 14:51:50.495584 5497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:51:50.492170 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:46497 (local address 127.0.0.1:50554)
0927 14:51:50.492670 (+ 500us) negotiation.cc:107] Waiting for socket to connect
0927 14:51:50.492713 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0927 14:51:50.492808 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:51:50.494196 (+ 1388us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:51:50.494201 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:51:50.494223 (+ 22us) client_negotiation.cc:189] Negotiated authn=SASL
0927 14:51:50.494822 (+ 599us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 14:51:50.494824 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0927 14:51:50.494835 (+ 11us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 14:51:50.494896 (+ 61us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 14:51:50.495308 (+ 412us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 14:51:50.495311 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 14:51:50.495314 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 14:51:50.495386 (+ 72us) client_negotiation.cc:240] Negotiation successful
0927 14:51:50.495421 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":335,"thread_start_us":74,"threads_started":1}
I20260927 14:51:50.496215 5501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:51:50.493325 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45796 (local address 127.0.98.3:40405)
0927 14:51:50.493723 (+ 398us) server_negotiation.cc:206] Beginning negotiation
0927 14:51:50.493730 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 14:51:50.493749 (+ 19us) server_negotiation.cc:407] Connection header received
0927 14:51:50.493791 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:51:50.493795 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:51:50.493931 (+ 136us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:51:50.494074 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:51:50.495046 (+ 972us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:51:50.495051 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:51:50.495061 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 14:51:50.495070 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:51:50.495105 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 14:51:50.495124 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:51:50.495239 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0927 14:51:50.495474 (+ 235us) server_negotiation.cc:299] Negotiation successful
0927 14:51:50.495547 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":322,"thread_start_us":100,"threads_started":1}
I20260927 14:51:50.496731 5496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:51:50.492285 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37973 (local address 127.0.0.1:39896)
0927 14:51:50.493274 (+ 989us) negotiation.cc:107] Waiting for socket to connect
0927 14:51:50.493288 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 14:51:50.493379 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:51:50.494668 (+ 1289us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:51:50.494671 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:51:50.494683 (+ 12us) client_negotiation.cc:189] Negotiated authn=SASL
0927 14:51:50.495312 (+ 629us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 14:51:50.495316 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 14:51:50.495327 (+ 11us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 14:51:50.495351 (+ 24us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 14:51:50.496511 (+ 1160us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 14:51:50.496515 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 14:51:50.496518 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 14:51:50.496585 (+ 67us) client_negotiation.cc:240] Negotiation successful
0927 14:51:50.496618 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":920,"thread_start_us":129,"threads_started":1}
I20260927 14:51:50.496984 5499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:51:50.492848 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39896 (local address 127.0.98.1:37973)
0927 14:51:50.493864 (+ 1016us) server_negotiation.cc:206] Beginning negotiation
0927 14:51:50.493878 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 14:51:50.493902 (+ 24us) server_negotiation.cc:407] Connection header received
0927 14:51:50.493966 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:51:50.493971 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:51:50.494137 (+ 166us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:51:50.494267 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:51:50.496071 (+ 1804us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:51:50.496153 (+ 82us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:51:50.496167 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 14:51:50.496183 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:51:50.496227 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 14:51:50.496415 (+ 188us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:51:50.496543 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0927 14:51:50.496595 (+ 52us) server_negotiation.cc:299] Negotiation successful
0927 14:51:50.496657 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":901,"thread_start_us":147,"threads_started":1}
I20260927 14:51:50.499082 5419 tablet_service.cc:1467] Processing CreateTablet for tablet e71538c5ca5942329dcb20ea163d7b6a (DEFAULT_TABLE table=test-table [id=a0e0903e285648e1aa4e46d462c6c17c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 14:51:50.499425 5157 tablet_service.cc:1467] Processing CreateTablet for tablet e71538c5ca5942329dcb20ea163d7b6a (DEFAULT_TABLE table=test-table [id=a0e0903e285648e1aa4e46d462c6c17c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 14:51:50.500135 5419 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e71538c5ca5942329dcb20ea163d7b6a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 14:51:50.500295 5157 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e71538c5ca5942329dcb20ea163d7b6a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 14:51:50.502985 5288 tablet_service.cc:1467] Processing CreateTablet for tablet e71538c5ca5942329dcb20ea163d7b6a (DEFAULT_TABLE table=test-table [id=a0e0903e285648e1aa4e46d462c6c17c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 14:51:50.504019 5288 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e71538c5ca5942329dcb20ea163d7b6a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 14:51:50.506368 5502 tablet_bootstrap.cc:492] T e71538c5ca5942329dcb20ea163d7b6a P b93265bf78594e7392160e7bbe2a6df8: Bootstrap starting.
I20260927 14:51:50.507740 5503 tablet_bootstrap.cc:492] T e71538c5ca5942329dcb20ea163d7b6a P aa9a06d24d58498bafdc611d5777e5cf: Bootstrap starting.
I20260927 14:51:50.509107 5502 tablet_bootstrap.cc:654] T e71538c5ca5942329dcb20ea163d7b6a P b93265bf78594e7392160e7bbe2a6df8: Neither blocks nor log segments found. Creating new log.
I20260927 14:51:50.509873 5503 tablet_bootstrap.cc:654] T e71538c5ca5942329dcb20ea163d7b6a P aa9a06d24d58498bafdc611d5777e5cf: Neither blocks nor log segments found. Creating new log.
I20260927 14:51:50.509858 5502 log.cc:824] T e71538c5ca5942329dcb20ea163d7b6a P b93265bf78594e7392160e7bbe2a6df8: Log is configured to *not* fsync() on all Append() calls
I20260927 14:51:50.510664 5503 log.cc:824] T e71538c5ca5942329dcb20ea163d7b6a P aa9a06d24d58498bafdc611d5777e5cf: Log is configured to *not* fsync() on all Append() calls
I20260927 14:51:50.511071 5505 tablet_bootstrap.cc:492] T e71538c5ca5942329dcb20ea163d7b6a P 2ee1ebb2fee64a9a98469148f2d7e0a2: Bootstrap starting.
I20260927 14:51:50.511925 5502 tablet_bootstrap.cc:492] T e71538c5ca5942329dcb20ea163d7b6a P b93265bf78594e7392160e7bbe2a6df8: No bootstrap required, opened a new log
I20260927 14:51:50.512121 5502 ts_tablet_manager.cc:1397] T e71538c5ca5942329dcb20ea163d7b6a P b93265bf78594e7392160e7bbe2a6df8: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260927 14:51:50.513219 5505 tablet_bootstrap.cc:654] T e71538c5ca5942329dcb20ea163d7b6a P 2ee1ebb2fee64a9a98469148f2d7e0a2: Neither blocks nor log segments found. Creating new log.
I20260927 14:51:50.514036 5505 log.cc:824] T e71538c5ca5942329dcb20ea163d7b6a P 2ee1ebb2fee64a9a98469148f2d7e0a2: Log is configured to *not* fsync() on all Append() calls
I20260927 14:51:50.514812 5503 tablet_bootstrap.cc:492] T e71538c5ca5942329dcb20ea163d7b6a P aa9a06d24d58498bafdc611d5777e5cf: No bootstrap required, opened a new log
I20260927 14:51:50.515027 5503 ts_tablet_manager.cc:1397] T e71538c5ca5942329dcb20ea163d7b6a P aa9a06d24d58498bafdc611d5777e5cf: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260927 14:51:50.516130 5505 tablet_bootstrap.cc:492] T e71538c5ca5942329dcb20ea163d7b6a P 2ee1ebb2fee64a9a98469148f2d7e0a2: No bootstrap required, opened a new log
I20260927 14:51:50.516310 5505 ts_tablet_manager.cc:1397] T e71538c5ca5942329dcb20ea163d7b6a P 2ee1ebb2fee64a9a98469148f2d7e0a2: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.004s
I20260927 14:51:50.516732 5502 raft_consensus.cc:348] T e71538c5ca5942329dcb20ea163d7b6a P b93265bf78594e7392160e7bbe2a6df8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ee1ebb2fee64a9a98469148f2d7e0a2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46497 } } peers { permanent_uuid: "aa9a06d24d58498bafdc611d5777e5cf" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37973 } } peers { permanent_uuid: "b93265bf78594e7392160e7bbe2a6df8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40405 } }
I20260927 14:51:50.517015 5502 raft_consensus.cc:374] T e71538c5ca5942329dcb20ea163d7b6a P b93265bf78594e7392160e7bbe2a6df8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 14:51:50.517098 5502 raft_consensus.cc:729] T e71538c5ca5942329dcb20ea163d7b6a P b93265bf78594e7392160e7bbe2a6df8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b93265bf78594e7392160e7bbe2a6df8, State: Initialized, Role: FOLLOWER
I20260927 14:51:50.517575 5502 consensus_queue.cc:260] T e71538c5ca5942329dcb20ea163d7b6a P b93265bf78594e7392160e7bbe2a6df8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ee1ebb2fee64a9a98469148f2d7e0a2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46497 } } peers { permanent_uuid: "aa9a06d24d58498bafdc611d5777e5cf" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37973 } } peers { permanent_uuid: "b93265bf78594e7392160e7bbe2a6df8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40405 } }
I20260927 14:51:50.518625 5502 ts_tablet_manager.cc:1428] T e71538c5ca5942329dcb20ea163d7b6a P b93265bf78594e7392160e7bbe2a6df8: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260927 14:51:50.518630 5503 raft_consensus.cc:348] T e71538c5ca5942329dcb20ea163d7b6a P aa9a06d24d58498bafdc611d5777e5cf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ee1ebb2fee64a9a98469148f2d7e0a2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46497 } } peers { permanent_uuid: "aa9a06d24d58498bafdc611d5777e5cf" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37973 } } peers { permanent_uuid: "b93265bf78594e7392160e7bbe2a6df8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40405 } }
I20260927 14:51:50.518887 5503 raft_consensus.cc:374] T e71538c5ca5942329dcb20ea163d7b6a P aa9a06d24d58498bafdc611d5777e5cf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 14:51:50.518952 5503 raft_consensus.cc:729] T e71538c5ca5942329dcb20ea163d7b6a P aa9a06d24d58498bafdc611d5777e5cf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aa9a06d24d58498bafdc611d5777e5cf, State: Initialized, Role: FOLLOWER
I20260927 14:51:50.519016 5485 heartbeater.cc:503] Master 127.0.98.62:37483 was elected leader, sending a full tablet report...
I20260927 14:51:50.519345 5503 consensus_queue.cc:260] T e71538c5ca5942329dcb20ea163d7b6a P aa9a06d24d58498bafdc611d5777e5cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ee1ebb2fee64a9a98469148f2d7e0a2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46497 } } peers { permanent_uuid: "aa9a06d24d58498bafdc611d5777e5cf" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37973 } } peers { permanent_uuid: "b93265bf78594e7392160e7bbe2a6df8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40405 } }
I20260927 14:51:50.520247 5503 ts_tablet_manager.cc:1428] T e71538c5ca5942329dcb20ea163d7b6a P aa9a06d24d58498bafdc611d5777e5cf: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260927 14:51:50.520553 5223 heartbeater.cc:503] Master 127.0.98.62:37483 was elected leader, sending a full tablet report...
I20260927 14:51:50.521607 5505 raft_consensus.cc:348] T e71538c5ca5942329dcb20ea163d7b6a P 2ee1ebb2fee64a9a98469148f2d7e0a2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ee1ebb2fee64a9a98469148f2d7e0a2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46497 } } peers { permanent_uuid: "aa9a06d24d58498bafdc611d5777e5cf" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37973 } } peers { permanent_uuid: "b93265bf78594e7392160e7bbe2a6df8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40405 } }
I20260927 14:51:50.521886 5505 raft_consensus.cc:374] T e71538c5ca5942329dcb20ea163d7b6a P 2ee1ebb2fee64a9a98469148f2d7e0a2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 14:51:50.521947 5505 raft_consensus.cc:729] T e71538c5ca5942329dcb20ea163d7b6a P 2ee1ebb2fee64a9a98469148f2d7e0a2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2ee1ebb2fee64a9a98469148f2d7e0a2, State: Initialized, Role: FOLLOWER
I20260927 14:51:50.522367 5505 consensus_queue.cc:260] T e71538c5ca5942329dcb20ea163d7b6a P 2ee1ebb2fee64a9a98469148f2d7e0a2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ee1ebb2fee64a9a98469148f2d7e0a2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46497 } } peers { permanent_uuid: "aa9a06d24d58498bafdc611d5777e5cf" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37973 } } peers { permanent_uuid: "b93265bf78594e7392160e7bbe2a6df8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40405 } }
I20260927 14:51:50.523075 5354 heartbeater.cc:503] Master 127.0.98.62:37483 was elected leader, sending a full tablet report...
I20260927 14:51:50.523306 5505 ts_tablet_manager.cc:1428] T e71538c5ca5942329dcb20ea163d7b6a P 2ee1ebb2fee64a9a98469148f2d7e0a2: Time spent starting tablet: real 0.007s user 0.005s sys 0.002s
W20260927 14:51:50.533345 5355 tablet.cc:2369] T e71538c5ca5942329dcb20ea163d7b6a P 2ee1ebb2fee64a9a98469148f2d7e0a2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 14:51:50.594887 5510 raft_consensus.cc:482] T e71538c5ca5942329dcb20ea163d7b6a P 2ee1ebb2fee64a9a98469148f2d7e0a2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 14:51:50.595129 5510 raft_consensus.cc:504] T e71538c5ca5942329dcb20ea163d7b6a P 2ee1ebb2fee64a9a98469148f2d7e0a2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ee1ebb2fee64a9a98469148f2d7e0a2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46497 } } peers { permanent_uuid: "aa9a06d24d58498bafdc611d5777e5cf" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37973 } } peers { permanent_uuid: "b93265bf78594e7392160e7bbe2a6df8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40405 } }
I20260927 14:51:50.596257 5510 leader_election.cc:288] T e71538c5ca5942329dcb20ea163d7b6a P 2ee1ebb2fee64a9a98469148f2d7e0a2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers aa9a06d24d58498bafdc611d5777e5cf (127.0.98.1:37973), b93265bf78594e7392160e7bbe2a6df8 (127.0.98.3:40405)
I20260927 14:51:50.598356 5356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:51:50.596537 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37973 (local address 127.0.98.2:47945)
0927 14:51:50.596765 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0927 14:51:50.596780 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 14:51:50.596890 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:51:50.597174 (+ 284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:51:50.597177 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:51:50.597191 (+ 14us) client_negotiation.cc:189] Negotiated authn=SASL
0927 14:51:50.597752 (+ 561us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 14:51:50.597755 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0927 14:51:50.597768 (+ 13us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 14:51:50.597802 (+ 34us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 14:51:50.598078 (+ 276us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 14:51:50.598082 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 14:51:50.598089 (+ 7us) client_negotiation.cc:769] Sending connection context
0927 14:51:50.598174 (+ 85us) client_negotiation.cc:240] Negotiation successful
0927 14:51:50.598204 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":153}
I20260927 14:51:50.598382 5499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:51:50.596594 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:47945 (local address 127.0.98.1:37973)
0927 14:51:50.596728 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0927 14:51:50.596733 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 14:51:50.596876 (+ 143us) server_negotiation.cc:407] Connection header received
0927 14:51:50.597010 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:51:50.597016 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:51:50.597079 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:51:50.597157 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:51:50.597940 (+ 783us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:51:50.597946 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:51:50.597950 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 14:51:50.597963 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:51:50.597986 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 14:51:50.598001 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:51:50.598092 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0927 14:51:50.598179 (+ 87us) server_negotiation.cc:299] Negotiation successful
0927 14:51:50.598224 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260927 14:51:50.598526 5511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:51:50.596565 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:40405 (local address 127.0.98.2:43289)
0927 14:51:50.596928 (+ 363us) negotiation.cc:107] Waiting for socket to connect
0927 14:51:50.596958 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0927 14:51:50.597087 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:51:50.597327 (+ 240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:51:50.597330 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:51:50.597344 (+ 14us) client_negotiation.cc:189] Negotiated authn=SASL
0927 14:51:50.597980 (+ 636us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 14:51:50.597981 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0927 14:51:50.597990 (+ 9us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 14:51:50.598013 (+ 23us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 14:51:50.598292 (+ 279us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 14:51:50.598294 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 14:51:50.598297 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 14:51:50.598369 (+ 72us) client_negotiation.cc:240] Negotiation successful
0927 14:51:50.598404 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":284,"thread_start_us":129,"threads_started":1}
I20260927 14:51:50.598527 5501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:51:50.596647 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:43289 (local address 127.0.98.3:40405)
0927 14:51:50.596869 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0927 14:51:50.596874 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 14:51:50.597048 (+ 174us) server_negotiation.cc:407] Connection header received
0927 14:51:50.597171 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:51:50.597173 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:51:50.597209 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:51:50.597293 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:51:50.598155 (+ 862us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:51:50.598160 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:51:50.598165 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 14:51:50.598176 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:51:50.598197 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 14:51:50.598208 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:51:50.598289 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0927 14:51:50.598375 (+ 86us) server_negotiation.cc:299] Negotiation successful
0927 14:51:50.598416 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":120}
I20260927 14:51:50.599099 5177 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e71538c5ca5942329dcb20ea163d7b6a" candidate_uuid: "2ee1ebb2fee64a9a98469148f2d7e0a2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aa9a06d24d58498bafdc611d5777e5cf" is_pre_election: true
I20260927 14:51:50.599110 5439 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e71538c5ca5942329dcb20ea163d7b6a" candidate_uuid: "2ee1ebb2fee64a9a98469148f2d7e0a2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b93265bf78594e7392160e7bbe2a6df8" is_pre_election: true
I20260927 14:51:50.599378 5439 raft_consensus.cc:2454] T e71538c5ca5942329dcb20ea163d7b6a P b93265bf78594e7392160e7bbe2a6df8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2ee1ebb2fee64a9a98469148f2d7e0a2 in term 0.
I20260927 14:51:50.599381 5177 raft_consensus.cc:2454] T e71538c5ca5942329dcb20ea163d7b6a P aa9a06d24d58498bafdc611d5777e5cf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2ee1ebb2fee64a9a98469148f2d7e0a2 in term 0.
I20260927 14:51:50.599822 5242 leader_election.cc:302] T e71538c5ca5942329dcb20ea163d7b6a P 2ee1ebb2fee64a9a98469148f2d7e0a2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2ee1ebb2fee64a9a98469148f2d7e0a2, aa9a06d24d58498bafdc611d5777e5cf; no voters:
I20260927 14:51:50.600069 5510 raft_consensus.cc:2781] T e71538c5ca5942329dcb20ea163d7b6a P 2ee1ebb2fee64a9a98469148f2d7e0a2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 14:51:50.600157 5510 raft_consensus.cc:482] T e71538c5ca5942329dcb20ea163d7b6a P 2ee1ebb2fee64a9a98469148f2d7e0a2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 14:51:50.600217 5510 raft_consensus.cc:3037] T e71538c5ca5942329dcb20ea163d7b6a P 2ee1ebb2fee64a9a98469148f2d7e0a2 [term 0 FOLLOWER]: Advancing to term 1
I20260927 14:51:50.601318 5510 raft_consensus.cc:504] T e71538c5ca5942329dcb20ea163d7b6a P 2ee1ebb2fee64a9a98469148f2d7e0a2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ee1ebb2fee64a9a98469148f2d7e0a2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46497 } } peers { permanent_uuid: "aa9a06d24d58498bafdc611d5777e5cf" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37973 } } peers { permanent_uuid: "b93265bf78594e7392160e7bbe2a6df8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40405 } }
I20260927 14:51:50.601668 5510 leader_election.cc:288] T e71538c5ca5942329dcb20ea163d7b6a P 2ee1ebb2fee64a9a98469148f2d7e0a2 [CANDIDATE]: Term 1 election: Requested vote from peers aa9a06d24d58498bafdc611d5777e5cf (127.0.98.1:37973), b93265bf78594e7392160e7bbe2a6df8 (127.0.98.3:40405)
I20260927 14:51:50.602032 5439 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e71538c5ca5942329dcb20ea163d7b6a" candidate_uuid: "2ee1ebb2fee64a9a98469148f2d7e0a2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b93265bf78594e7392160e7bbe2a6df8"
I20260927 14:51:50.602073 5177 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e71538c5ca5942329dcb20ea163d7b6a" candidate_uuid: "2ee1ebb2fee64a9a98469148f2d7e0a2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aa9a06d24d58498bafdc611d5777e5cf"
I20260927 14:51:50.602144 5439 raft_consensus.cc:3037] T e71538c5ca5942329dcb20ea163d7b6a P b93265bf78594e7392160e7bbe2a6df8 [term 0 FOLLOWER]: Advancing to term 1
I20260927 14:51:50.602208 5177 raft_consensus.cc:3037] T e71538c5ca5942329dcb20ea163d7b6a P aa9a06d24d58498bafdc611d5777e5cf [term 0 FOLLOWER]: Advancing to term 1
I20260927 14:51:50.603190 5439 raft_consensus.cc:2454] T e71538c5ca5942329dcb20ea163d7b6a P b93265bf78594e7392160e7bbe2a6df8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2ee1ebb2fee64a9a98469148f2d7e0a2 in term 1.
I20260927 14:51:50.603193 5177 raft_consensus.cc:2454] T e71538c5ca5942329dcb20ea163d7b6a P aa9a06d24d58498bafdc611d5777e5cf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2ee1ebb2fee64a9a98469148f2d7e0a2 in term 1.
I20260927 14:51:50.603466 5242 leader_election.cc:302] T e71538c5ca5942329dcb20ea163d7b6a P 2ee1ebb2fee64a9a98469148f2d7e0a2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2ee1ebb2fee64a9a98469148f2d7e0a2, aa9a06d24d58498bafdc611d5777e5cf; no voters:
I20260927 14:51:50.603659 5510 raft_consensus.cc:2781] T e71538c5ca5942329dcb20ea163d7b6a P 2ee1ebb2fee64a9a98469148f2d7e0a2 [term 1 FOLLOWER]: Leader election won for term 1
W20260927 14:51:50.603909 5224 tablet.cc:2369] T e71538c5ca5942329dcb20ea163d7b6a P aa9a06d24d58498bafdc611d5777e5cf: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 14:51:50.603955 5510 raft_consensus.cc:686] T e71538c5ca5942329dcb20ea163d7b6a P 2ee1ebb2fee64a9a98469148f2d7e0a2 [term 1 LEADER]: Becoming Leader. State: Replica: 2ee1ebb2fee64a9a98469148f2d7e0a2, State: Running, Role: LEADER
I20260927 14:51:50.604269 5510 consensus_queue.cc:237] T e71538c5ca5942329dcb20ea163d7b6a P 2ee1ebb2fee64a9a98469148f2d7e0a2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ee1ebb2fee64a9a98469148f2d7e0a2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46497 } } peers { permanent_uuid: "aa9a06d24d58498bafdc611d5777e5cf" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37973 } } peers { permanent_uuid: "b93265bf78594e7392160e7bbe2a6df8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40405 } }
I20260927 14:51:50.607354 5035 catalog_manager.cc:5495] T e71538c5ca5942329dcb20ea163d7b6a P 2ee1ebb2fee64a9a98469148f2d7e0a2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2ee1ebb2fee64a9a98469148f2d7e0a2 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "2ee1ebb2fee64a9a98469148f2d7e0a2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ee1ebb2fee64a9a98469148f2d7e0a2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46497 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "aa9a06d24d58498bafdc611d5777e5cf" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37973 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b93265bf78594e7392160e7bbe2a6df8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40405 } health_report { overall_health: UNKNOWN } } }
I20260927 14:51:50.617403 5498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:51:50.615579 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50556 (local address 127.0.98.2:46497)
0927 14:51:50.615722 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0927 14:51:50.615729 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 14:51:50.615748 (+ 19us) server_negotiation.cc:407] Connection header received
0927 14:51:50.615991 (+ 243us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:51:50.615994 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:51:50.616054 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:51:50.616144 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:51:50.616896 (+ 752us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:51:50.616900 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:51:50.616903 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 14:51:50.616917 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:51:50.616938 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 14:51:50.616956 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:51:50.617060 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0927 14:51:50.617190 (+ 130us) server_negotiation.cc:299] Negotiation successful
0927 14:51:50.617233 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260927 14:51:50.623755 5177 raft_consensus.cc:1261] T e71538c5ca5942329dcb20ea163d7b6a P aa9a06d24d58498bafdc611d5777e5cf [term 1 FOLLOWER]: Refusing update from remote peer 2ee1ebb2fee64a9a98469148f2d7e0a2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 14:51:50.623754 5439 raft_consensus.cc:1261] T e71538c5ca5942329dcb20ea163d7b6a P b93265bf78594e7392160e7bbe2a6df8 [term 1 FOLLOWER]: Refusing update from remote peer 2ee1ebb2fee64a9a98469148f2d7e0a2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 14:51:50.624595 5512 consensus_queue.cc:1035] T e71538c5ca5942329dcb20ea163d7b6a P 2ee1ebb2fee64a9a98469148f2d7e0a2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aa9a06d24d58498bafdc611d5777e5cf" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37973 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 14:51:50.624871 5510 consensus_queue.cc:1035] T e71538c5ca5942329dcb20ea163d7b6a P 2ee1ebb2fee64a9a98469148f2d7e0a2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b93265bf78594e7392160e7bbe2a6df8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40405 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 14:51:50.632237 5517 mvcc.cc:205] Tried to move back new op lower bound from 7333972830706806784 to 7333972830637424640. Current Snapshot: MvccSnapshot[applied={T|T < 7333972830706806784}]
I20260927 14:51:50.640753 5035 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:44106:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 14:51:50.640923 5035 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:44106:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 14:51:50.643299 5035 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 67a82d6810144c32be0112f256962174: Sending DeleteTablet for 3 replicas of tablet e71538c5ca5942329dcb20ea163d7b6a
I20260927 14:51:50.644027 5288 tablet_service.cc:1514] Processing DeleteTablet for tablet e71538c5ca5942329dcb20ea163d7b6a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 14:51:50 UTC) from {username='slave'} at 127.0.0.1:50554
I20260927 14:51:50.644069 5157 tablet_service.cc:1514] Processing DeleteTablet for tablet e71538c5ca5942329dcb20ea163d7b6a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 14:51:50 UTC) from {username='slave'} at 127.0.0.1:39896
I20260927 14:51:50.644490 5525 tablet_replica.cc:330] T e71538c5ca5942329dcb20ea163d7b6a P 2ee1ebb2fee64a9a98469148f2d7e0a2: stopping tablet replica
I20260927 14:51:50.644855 5525 raft_consensus.cc:2229] T e71538c5ca5942329dcb20ea163d7b6a P 2ee1ebb2fee64a9a98469148f2d7e0a2 [term 1 LEADER]: Raft consensus shutting down.
I20260927 14:51:50.644805 5419 tablet_service.cc:1514] Processing DeleteTablet for tablet e71538c5ca5942329dcb20ea163d7b6a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 14:51:50 UTC) from {username='slave'} at 127.0.0.1:45796
I20260927 14:51:50.645192 5526 tablet_replica.cc:330] T e71538c5ca5942329dcb20ea163d7b6a P aa9a06d24d58498bafdc611d5777e5cf: stopping tablet replica
I20260927 14:51:50.645354 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6936m5/build/debug/bin/kudu with pid 5095
I20260927 14:51:50.645480 5526 raft_consensus.cc:2229] T e71538c5ca5942329dcb20ea163d7b6a P aa9a06d24d58498bafdc611d5777e5cf [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 14:51:50.645727 5526 raft_consensus.cc:2258] T e71538c5ca5942329dcb20ea163d7b6a P aa9a06d24d58498bafdc611d5777e5cf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 14:51:50.645381 5525 raft_consensus.cc:2258] T e71538c5ca5942329dcb20ea163d7b6a P 2ee1ebb2fee64a9a98469148f2d7e0a2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 14:51:50.648698 5525 ts_tablet_manager.cc:1905] T e71538c5ca5942329dcb20ea163d7b6a P 2ee1ebb2fee64a9a98469148f2d7e0a2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 14:51:50.651659 5527 tablet_replica.cc:330] T e71538c5ca5942329dcb20ea163d7b6a P b93265bf78594e7392160e7bbe2a6df8: stopping tablet replica
I20260927 14:51:50.652004 5527 raft_consensus.cc:2229] T e71538c5ca5942329dcb20ea163d7b6a P b93265bf78594e7392160e7bbe2a6df8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 14:51:50.652290 5525 ts_tablet_manager.cc:1918] T e71538c5ca5942329dcb20ea163d7b6a P 2ee1ebb2fee64a9a98469148f2d7e0a2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 14:51:50.652344 5527 raft_consensus.cc:2258] T e71538c5ca5942329dcb20ea163d7b6a P b93265bf78594e7392160e7bbe2a6df8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 14:51:50.652406 5525 log.cc:1196] T e71538c5ca5942329dcb20ea163d7b6a P 2ee1ebb2fee64a9a98469148f2d7e0a2: Deleting WAL directory at /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790520684602251-392-0/minicluster-data/ts-1/wal/wals/e71538c5ca5942329dcb20ea163d7b6a
I20260927 14:51:50.652736 5525 ts_tablet_manager.cc:1939] T e71538c5ca5942329dcb20ea163d7b6a P 2ee1ebb2fee64a9a98469148f2d7e0a2: Deleting consensus metadata
W20260927 14:51:50.653249 5020 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:37973 (error 108)
I20260927 14:51:50.653892 5022 catalog_manager.cc:4841] TS 2ee1ebb2fee64a9a98469148f2d7e0a2 (127.0.98.2:46497): tablet e71538c5ca5942329dcb20ea163d7b6a (table test-table [id=a0e0903e285648e1aa4e46d462c6c17c]) successfully deleted
I20260927 14:51:50.654290 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6936m5/build/debug/bin/kudu with pid 5226
I20260927 14:51:50.654649 5496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:51:50.654318 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37973 (local address 127.0.0.1:39902)
0927 14:51:50.654488 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0927 14:51:50.654558 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:37973: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
W20260927 14:51:50.654909 5020 catalog_manager.cc:4584] TS aa9a06d24d58498bafdc611d5777e5cf (127.0.98.1:37973): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet e71538c5ca5942329dcb20ea163d7b6a: Network error: Client connection negotiation failed: client connection to 127.0.98.1:37973: connect: Connection refused (error 111)
I20260927 14:51:50.654160 5527 ts_tablet_manager.cc:1905] T e71538c5ca5942329dcb20ea163d7b6a P b93265bf78594e7392160e7bbe2a6df8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 14:51:50.658883 5527 ts_tablet_manager.cc:1918] T e71538c5ca5942329dcb20ea163d7b6a P b93265bf78594e7392160e7bbe2a6df8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 14:51:50.658985 5527 log.cc:1196] T e71538c5ca5942329dcb20ea163d7b6a P b93265bf78594e7392160e7bbe2a6df8: Deleting WAL directory at /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790520684602251-392-0/minicluster-data/ts-2/wal/wals/e71538c5ca5942329dcb20ea163d7b6a
I20260927 14:51:50.659288 5527 ts_tablet_manager.cc:1939] T e71538c5ca5942329dcb20ea163d7b6a P b93265bf78594e7392160e7bbe2a6df8: Deleting consensus metadata
I20260927 14:51:50.660563 5021 catalog_manager.cc:4841] TS b93265bf78594e7392160e7bbe2a6df8 (127.0.98.3:40405): tablet e71538c5ca5942329dcb20ea163d7b6a (table test-table [id=a0e0903e285648e1aa4e46d462c6c17c]) successfully deleted
I20260927 14:51:50.662112 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6936m5/build/debug/bin/kudu with pid 5357
I20260927 14:51:50.667384 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6936m5/build/debug/bin/kudu with pid 5004
2026-09-27T14:51:50Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (1050 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260927 14:51:50.688721 392 mini_oidc.cc:152] Starting JWKS server
I20260927 14:51:50.691084 392 webserver.cc:460] Webserver started at https://127.0.0.1:41941/ using document root <none> and password file <none>
I20260927 14:51:50.691290 392 mini_oidc.cc:189] Starting OIDC Discovery server
I20260927 14:51:50.691625 392 webserver.cc:460] Webserver started at http://127.0.0.1:33889/ using document root <none> and password file <none>
2026-09-27T14:51:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T14:51:50Z Disabled control of system clock
I20260927 14:51:50.700687 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6936m5/build/debug/bin/kudu
/tmp/dist-test-task6936m5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790520684602251-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790520684602251-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790520684602251-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790520684602251-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33777
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:45489
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:33777
--enable_jwt_token_auth=true
--jwks_url=https://localhost:41941/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790520684602251-392-0/testchainca.cert with env {}
W20260927 14:51:50.810650 5534 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 14:51:50.810933 5534 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 14:51:50.810976 5534 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 14:51:50.815217 5534 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 14:51:50.815289 5534 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 14:51:50.815337 5534 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 14:51:50.815361 5534 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 14:51:50.815390 5534 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260927 14:51:50.815428 5534 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260927 14:51:50.815449 5534 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:41941/jwks.json
I20260927 14:51:50.820950 5534 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45489
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790520684602251-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790520684602251-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:33777
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33777
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:41941/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790520684602251-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790520684602251-392-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5534
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790520684602251-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:15:17 UTC on ccycloud.quasar-ledxfe.root.comops.site
build id 4676134
I20260927 14:51:50.822052 5534 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 14:51:50.822963 5534 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 14:51:50.831301 5542 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 14:51:50.831352 5534 server_base.cc:1034] running on GCE node
W20260927 14:51:50.831398 5540 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 14:51:50.831382 5539 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 14:51:50.832161 5534 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 14:51:50.833274 5534 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 14:51:50.834474 5534 hybrid_clock.cc:648] HybridClock initialized: now 1790520710834452 us; error 33 us; skew 500 ppm
I20260927 14:51:50.836315 5534 webserver.cc:460] Webserver started at http://127.0.98.62:37263/ using document root <none> and password file <none>
I20260927 14:51:50.836913 5534 fs_manager.cc:359] Metadata directory not provided
I20260927 14:51:50.837023 5534 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 14:51:50.837268 5534 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 14:51:50.839265 5534 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790520684602251-392-0/minicluster-data/master-0/data/instance:
uuid: "f24ad56be96a499986b1c1aed2346231"
format_stamp: "Formatted at 2026-09-27 14:51:50 on dist-test-slave-xk1g"
server_key: "5a9d97dd7f54abfe6fa52d820b5907cf"
server_key_iv: "e6cb2b73380f982c9f2f86c6820c3c96"
server_key_version: "encryptionkey@0"
I20260927 14:51:50.839824 5534 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790520684602251-392-0/minicluster-data/master-0/wal/instance:
uuid: "f24ad56be96a499986b1c1aed2346231"
format_stamp: "Formatted at 2026-09-27 14:51:50 on dist-test-slave-xk1g"
server_key: "5a9d97dd7f54abfe6fa52d820b5907cf"
server_key_iv: "e6cb2b73380f982c9f2f86c6820c3c96"
server_key_version: "encryptionkey@0"
I20260927 14:51:50.843499 5534 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.002s
I20260927 14:51:50.846076 5548 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 14:51:50.847224 5534 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 14:51:50.847376 5534 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790520684602251-392-0/minicluster-data/master-0/data,/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790520684602251-392-0/minicluster-data/master-0/wal
uuid: "f24ad56be96a499986b1c1aed2346231"
format_stamp: "Formatted at 2026-09-27 14:51:50 on dist-test-slave-xk1g"
server_key: "5a9d97dd7f54abfe6fa52d820b5907cf"
server_key_iv: "e6cb2b73380f982c9f2f86c6820c3c96"
server_key_version: "encryptionkey@0"
I20260927 14:51:50.847487 5534 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790520684602251-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790520684602251-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790520684602251-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 14:51:50.874918 5534 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 14:51:50.875658 5534 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 14:51:50.875864 5534 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 14:51:50.882668 5534 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:33777
I20260927 14:51:50.882673 5600 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:33777 every 8 connection(s)
I20260927 14:51:50.883651 5534 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790520684602251-392-0/minicluster-data/master-0/data/info.pb
I20260927 14:51:50.886750 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6936m5/build/debug/bin/kudu as pid 5534
I20260927 14:51:50.886878 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790520684602251-392-0/minicluster-data/master-0/wal/instance
I20260927 14:51:50.887197 392 external_mini_cluster.cc:1428] Setting key 70b7bdf7557e81d4458f07a821732de5
I20260927 14:51:50.887327 5601 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 14:51:50.894454 5601 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f24ad56be96a499986b1c1aed2346231: Bootstrap starting.
I20260927 14:51:50.896762 5604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:51:50.888654 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33704 (local address 127.0.98.62:33777)
0927 14:51:50.889385 (+ 731us) server_negotiation.cc:206] Beginning negotiation
0927 14:51:50.889400 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 14:51:50.889437 (+ 37us) server_negotiation.cc:407] Connection header received
0927 14:51:50.890091 (+ 654us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:51:50.890108 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:51:50.890497 (+ 389us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:51:50.890777 (+ 280us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:51:50.891107 (+ 330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:51:50.892153 (+ 1046us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:51:50.893454 (+ 1301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:51:50.893920 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:51:50.894043 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:51:50.894989 (+ 946us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:51:50.894994 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:51:50.895007 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 14:51:50.895415 (+ 408us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:51:50.895473 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 14:51:50.895501 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:51:50.895582 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0927 14:51:50.895756 (+ 174us) server_negotiation.cc:299] Negotiation successful
0927 14:51:50.896058 (+ 302us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":285,"thread_start_us":129,"threads_started":1}
I20260927 14:51:50.896926 5601 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f24ad56be96a499986b1c1aed2346231: Neither blocks nor log segments found. Creating new log.
I20260927 14:51:50.897677 5601 log.cc:824] T 00000000000000000000000000000000 P f24ad56be96a499986b1c1aed2346231: Log is configured to *not* fsync() on all Append() calls
I20260927 14:51:50.899718 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 14:51:50.899839 5601 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f24ad56be96a499986b1c1aed2346231: No bootstrap required, opened a new log
I20260927 14:51:50.903506 5601 raft_consensus.cc:348] T 00000000000000000000000000000000 P f24ad56be96a499986b1c1aed2346231 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f24ad56be96a499986b1c1aed2346231" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33777 } }
I20260927 14:51:50.903743 5601 raft_consensus.cc:374] T 00000000000000000000000000000000 P f24ad56be96a499986b1c1aed2346231 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 14:51:50.903786 5601 raft_consensus.cc:729] T 00000000000000000000000000000000 P f24ad56be96a499986b1c1aed2346231 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f24ad56be96a499986b1c1aed2346231, State: Initialized, Role: FOLLOWER
I20260927 14:51:50.904137 5601 consensus_queue.cc:260] T 00000000000000000000000000000000 P f24ad56be96a499986b1c1aed2346231 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f24ad56be96a499986b1c1aed2346231" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33777 } }
I20260927 14:51:50.904243 5601 raft_consensus.cc:388] T 00000000000000000000000000000000 P f24ad56be96a499986b1c1aed2346231 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 14:51:50.904321 5601 raft_consensus.cc:482] T 00000000000000000000000000000000 P f24ad56be96a499986b1c1aed2346231 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 14:51:50.904385 5601 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f24ad56be96a499986b1c1aed2346231 [term 0 FOLLOWER]: Advancing to term 1
I20260927 14:51:50.905251 5601 raft_consensus.cc:504] T 00000000000000000000000000000000 P f24ad56be96a499986b1c1aed2346231 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f24ad56be96a499986b1c1aed2346231" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33777 } }
I20260927 14:51:50.905534 5601 leader_election.cc:302] T 00000000000000000000000000000000 P f24ad56be96a499986b1c1aed2346231 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f24ad56be96a499986b1c1aed2346231; no voters:
I20260927 14:51:50.905781 5601 leader_election.cc:288] T 00000000000000000000000000000000 P f24ad56be96a499986b1c1aed2346231 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 14:51:50.905920 5606 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f24ad56be96a499986b1c1aed2346231 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 14:51:50.906142 5606 raft_consensus.cc:686] T 00000000000000000000000000000000 P f24ad56be96a499986b1c1aed2346231 [term 1 LEADER]: Becoming Leader. State: Replica: f24ad56be96a499986b1c1aed2346231, State: Running, Role: LEADER
I20260927 14:51:50.906483 5606 consensus_queue.cc:237] T 00000000000000000000000000000000 P f24ad56be96a499986b1c1aed2346231 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f24ad56be96a499986b1c1aed2346231" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33777 } }
I20260927 14:51:50.907058 5601 sys_catalog.cc:564] T 00000000000000000000000000000000 P f24ad56be96a499986b1c1aed2346231 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 14:51:50.908208 5608 sys_catalog.cc:455] T 00000000000000000000000000000000 P f24ad56be96a499986b1c1aed2346231 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f24ad56be96a499986b1c1aed2346231. Latest consensus state: current_term: 1 leader_uuid: "f24ad56be96a499986b1c1aed2346231" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f24ad56be96a499986b1c1aed2346231" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33777 } } }
I20260927 14:51:50.908210 5607 sys_catalog.cc:455] T 00000000000000000000000000000000 P f24ad56be96a499986b1c1aed2346231 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f24ad56be96a499986b1c1aed2346231" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f24ad56be96a499986b1c1aed2346231" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33777 } } }
I20260927 14:51:50.908366 5608 sys_catalog.cc:458] T 00000000000000000000000000000000 P f24ad56be96a499986b1c1aed2346231 [sys.catalog]: This master's current role is: LEADER
I20260927 14:51:50.908386 5607 sys_catalog.cc:458] T 00000000000000000000000000000000 P f24ad56be96a499986b1c1aed2346231 [sys.catalog]: This master's current role is: LEADER
I20260927 14:51:50.908758 5615 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 14:51:50.911852 5615 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 14:51:50.917271 5615 catalog_manager.cc:1347] Generated new cluster ID: f1a5a9c88be146e48b3c72c1b20e4900
I20260927 14:51:50.917335 5615 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 14:51:50.930296 5615 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 14:51:50.931247 5615 catalog_manager.cc:1504] Loading token signing keys...
I20260927 14:51:50.940773 5615 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f24ad56be96a499986b1c1aed2346231: Generated new TSK 0
I20260927 14:51:50.941569 5615 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260927 14:51:50.977118 5631 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260927 14:51:50.977246 5631 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260927 14:51:50.977823 5604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:51:50.960395 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33710 (local address 127.0.98.62:33777)
0927 14:51:50.960533 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0927 14:51:50.960538 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 14:51:50.960727 (+ 189us) server_negotiation.cc:407] Connection header received
0927 14:51:50.960931 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:51:50.960935 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:51:50.960994 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:51:50.961141 (+ 147us) server_negotiation.cc:226] Negotiated authn=JWT
0927 14:51:50.961434 (+ 293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:51:50.962244 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:51:50.963128 (+ 884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:51:50.963486 (+ 358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:51:50.963605 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:51:50.963953 (+ 348us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0927 14:51:50.977398 (+ 13445us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41941/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0927 14:51:50.977684 (+ 286us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:33710: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41941/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":53}
W20260927 14:51:50.977888 5604 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:33710: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41941/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260927 14:51:50.978973 5630 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 14:51:50.960263 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33777 (local address 127.0.0.1:33710)
0927 14:51:50.960643 (+ 380us) negotiation.cc:107] Waiting for socket to connect
0927 14:51:50.960656 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 14:51:50.960769 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:51:50.961188 (+ 419us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:51:50.961190 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:51:50.961198 (+ 8us) client_negotiation.cc:189] Negotiated authn=JWT
0927 14:51:50.961305 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:51:50.961309 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:51:50.962376 (+ 1067us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:51:50.962379 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:51:50.963016 (+ 637us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:51:50.963024 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:51:50.963619 (+ 595us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:51:50.963636 (+ 17us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:51:50.963713 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:51:50.963759 (+ 46us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0927 14:51:50.977653 (+ 13894us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41941/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0927 14:51:50.977864 (+ 211us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.98.62:33777: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41941/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":231,"thread_start_us":116,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.0.98.62:33777: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41941/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260927 14:51:50.979715 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6936m5/build/debug/bin/kudu with pid 5534
2026-09-27T14:51:51Z chronyd exiting
I20260927 14:51:51.016955 392 test_util.cc:175] -----------------------------------------------
I20260927 14:51:51.017043 392 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790520684602251-392-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (329 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260927 14:51:51.018131 392 mini_oidc.cc:152] Starting JWKS server
I20260927 14:51:51.018976 392 webserver.cc:460] Webserver started at https://127.0.0.1:40103/ using document root <none> and password file <none>
I20260927 14:51:51.019058 392 mini_oidc.cc:189] Starting OIDC Discovery server
I20260927 14:51:51.019265 392 webserver.cc:460] Webserver started at http://127.0.0.1:40439/ using document root <none> and password file <none>
2026-09-27T14:51:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T14:51:51Z Disabled control of system clock
I20260927 14:51:51.028182 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6936m5/build/debug/bin/kudu
/tmp/dist-test-task6936m5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790520684602251-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790520684602251-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790520684602251-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790520684602251-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37911
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:42317
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:37911
--enable_jwt_token_auth=true
--jwks_url=https://localhost:40103/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790520684602251-392-0/testchainca.cert with env {}
W20260927 14:51:51.135910 5637 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 14:51:51.136191 5637 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 14:51:51.136232 5637 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 14:51:51.140128 5637 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 14:51:51.140187 5637 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 14:51:51.140202 5637 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 14:51:51.140219 5637 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 14:51:51.140239 5637 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260927 14:51:51.140254 5637 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260927 14:51:51.140266 5637 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:40103/jwks.json
I20260927 14:51:51.145536 5637 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42317
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790520684602251-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790520684602251-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:37911
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37911
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:40103/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790520684602251-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790520684602251-392-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5637
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790520684602251-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:15:17 UTC on ccycloud.quasar-ledxfe.root.comops.site
build id 4676134
I20260927 14:51:51.146533 5637 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 14:51:51.147476 5637 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 14:51:51.155701 5643 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 14:51:51.155769 5642 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 14:51:51.155939 5637 server_base.cc:1034] running on GCE node
W20260927 14:51:51.155769 5645 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 14:51:51.156649 5637 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 14:51:51.157797 5637 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 14:51:51.158981 5637 hybrid_clock.cc:648] HybridClock initialized: now 1790520711158971 us; error 29 us; skew 500 ppm
I20260927 14:51:51.160737 5637 webserver.cc:460] Webserver started at http://127.0.98.62:43417/ using document root <none> and password file <none>
I20260927 14:51:51.161358 5637 fs_manager.cc:359] Metadata directory not provided
I20260927 14:51:51.161459 5637 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 14:51:51.161684 5637 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 14:51:51.163674 5637 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790520684602251-392-0/minicluster-data/master-0/data/instance:
uuid: "e4db9674660044f098f258c8d2608495"
format_stamp: "Formatted at 2026-09-27 14:51:51 on dist-test-slave-xk1g"
server_key: "38fefcd092a9a22610feacf14fcd8d10"
server_key_iv: "3554bc3b5a574e35ad706bc377d6ebee"
server_key_version: "encryptionkey@0"
I20260927 14:51:51.164232 5637 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790520684602251-392-0/minicluster-data/master-0/wal/instance:
uuid: "e4db9674660044f098f258c8d2608495"
format_stamp: "Formatted at 2026-09-27 14:51:51 on dist-test-slave-xk1g"
server_key: "38fefcd092a9a22610feacf14fcd8d10"
server_key_iv: "3554bc3b5a574e35ad706bc377d6ebee"
server_key_version: "encryptionkey@0"
I20260927 14:51:51.167802 5637 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260927 14:51:51.170408 5651 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 14:51:51.171571 5637 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 14:51:51.171712 5637 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790520684602251-392-0/minicluster-data/master-0/data,/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790520684602251-392-0/minicluster-data/master-0/wal
uuid: "e4db9674660044f098f258c8d2608495"
format_stamp: "Formatted at 2026-09-27 14:51:51 on dist-test-slave-xk1g"
server_key: "38fefcd092a9a22610feacf14fcd8d10"
server_key_iv: "3554bc3b5a574e35ad706bc377d6ebee"
server_key_version: "encryptionkey@0"
I20260927 14:51:51.171825 5637 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790520684602251-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790520684602251-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790520684602251-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 14:51:51.183597 5637 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 14:51:51.184298 5637 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 14:51:51.184501 5637 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 14:51:51.191454 5637 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:37911
I20260927 14:51:51.191483 5703 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:37911 every 8 connection(s)
I20260927 14:51:51.192377 5637 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790520684602251-392-0/minicluster-data/master-0/data/info.pb
I20260927 14:51:51.193601 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6936m5/build/debug/bin/kudu as pid 5637
I20260927 14:51:51.193732 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790520684602251-392-0/minicluster-data/master-0/wal/instance
I20260927 14:51:51.194000 392 external_mini_cluster.cc:1428] Setting key 12d4d6fab883880c3ad486db65e7a73a
I20260927 14:51:51.195871 5704 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 14:51:51.202235 5704 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e4db9674660044f098f258c8d2608495: Bootstrap starting.
I20260927 14:51:51.203016 5707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:51:51.195237 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36652 (local address 127.0.98.62:37911)
0927 14:51:51.195855 (+ 618us) server_negotiation.cc:206] Beginning negotiation
0927 14:51:51.195870 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 14:51:51.195906 (+ 36us) server_negotiation.cc:407] Connection header received
0927 14:51:51.196567 (+ 661us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:51:51.196583 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:51:51.196975 (+ 392us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:51:51.197324 (+ 349us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:51:51.197653 (+ 329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:51:51.198589 (+ 936us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:51:51.199824 (+ 1235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:51:51.200273 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:51:51.200388 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:51:51.201277 (+ 889us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:51:51.201282 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:51:51.201293 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 14:51:51.201665 (+ 372us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:51:51.201720 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 14:51:51.201738 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:51:51.201811 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0927 14:51:51.201978 (+ 167us) server_negotiation.cc:299] Negotiation successful
0927 14:51:51.202265 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":280,"thread_start_us":137,"threads_started":1}
I20260927 14:51:51.204707 5704 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e4db9674660044f098f258c8d2608495: Neither blocks nor log segments found. Creating new log.
I20260927 14:51:51.205603 5704 log.cc:824] T 00000000000000000000000000000000 P e4db9674660044f098f258c8d2608495: Log is configured to *not* fsync() on all Append() calls
I20260927 14:51:51.205999 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 14:51:51.207688 5704 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e4db9674660044f098f258c8d2608495: No bootstrap required, opened a new log
I20260927 14:51:51.211063 5704 raft_consensus.cc:348] T 00000000000000000000000000000000 P e4db9674660044f098f258c8d2608495 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e4db9674660044f098f258c8d2608495" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37911 } }
I20260927 14:51:51.211302 5704 raft_consensus.cc:374] T 00000000000000000000000000000000 P e4db9674660044f098f258c8d2608495 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 14:51:51.211385 5704 raft_consensus.cc:729] T 00000000000000000000000000000000 P e4db9674660044f098f258c8d2608495 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e4db9674660044f098f258c8d2608495, State: Initialized, Role: FOLLOWER
I20260927 14:51:51.211831 5704 consensus_queue.cc:260] T 00000000000000000000000000000000 P e4db9674660044f098f258c8d2608495 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e4db9674660044f098f258c8d2608495" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37911 } }
I20260927 14:51:51.211989 5704 raft_consensus.cc:388] T 00000000000000000000000000000000 P e4db9674660044f098f258c8d2608495 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 14:51:51.212074 5704 raft_consensus.cc:482] T 00000000000000000000000000000000 P e4db9674660044f098f258c8d2608495 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 14:51:51.212190 5704 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e4db9674660044f098f258c8d2608495 [term 0 FOLLOWER]: Advancing to term 1
I20260927 14:51:51.213110 5704 raft_consensus.cc:504] T 00000000000000000000000000000000 P e4db9674660044f098f258c8d2608495 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e4db9674660044f098f258c8d2608495" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37911 } }
I20260927 14:51:51.213454 5704 leader_election.cc:302] T 00000000000000000000000000000000 P e4db9674660044f098f258c8d2608495 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e4db9674660044f098f258c8d2608495; no voters:
I20260927 14:51:51.213759 5704 leader_election.cc:288] T 00000000000000000000000000000000 P e4db9674660044f098f258c8d2608495 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 14:51:51.213897 5709 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e4db9674660044f098f258c8d2608495 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 14:51:51.214155 5709 raft_consensus.cc:686] T 00000000000000000000000000000000 P e4db9674660044f098f258c8d2608495 [term 1 LEADER]: Becoming Leader. State: Replica: e4db9674660044f098f258c8d2608495, State: Running, Role: LEADER
I20260927 14:51:51.214440 5709 consensus_queue.cc:237] T 00000000000000000000000000000000 P e4db9674660044f098f258c8d2608495 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e4db9674660044f098f258c8d2608495" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37911 } }
I20260927 14:51:51.214989 5704 sys_catalog.cc:564] T 00000000000000000000000000000000 P e4db9674660044f098f258c8d2608495 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 14:51:51.216091 5710 sys_catalog.cc:455] T 00000000000000000000000000000000 P e4db9674660044f098f258c8d2608495 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e4db9674660044f098f258c8d2608495" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e4db9674660044f098f258c8d2608495" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37911 } } }
I20260927 14:51:51.216223 5710 sys_catalog.cc:458] T 00000000000000000000000000000000 P e4db9674660044f098f258c8d2608495 [sys.catalog]: This master's current role is: LEADER
I20260927 14:51:51.216411 5711 sys_catalog.cc:455] T 00000000000000000000000000000000 P e4db9674660044f098f258c8d2608495 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e4db9674660044f098f258c8d2608495. Latest consensus state: current_term: 1 leader_uuid: "e4db9674660044f098f258c8d2608495" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e4db9674660044f098f258c8d2608495" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37911 } } }
I20260927 14:51:51.216559 5711 sys_catalog.cc:458] T 00000000000000000000000000000000 P e4db9674660044f098f258c8d2608495 [sys.catalog]: This master's current role is: LEADER
I20260927 14:51:51.216621 5718 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 14:51:51.219826 5718 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 14:51:51.225530 5718 catalog_manager.cc:1347] Generated new cluster ID: 2886783415084656ae25084419f3ef9f
I20260927 14:51:51.225633 5718 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 14:51:51.244985 5718 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 14:51:51.246034 5718 catalog_manager.cc:1504] Loading token signing keys...
I20260927 14:51:51.255075 5718 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e4db9674660044f098f258c8d2608495: Generated new TSK 0
I20260927 14:51:51.255846 5718 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260927 14:51:51.284022 5734 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260927 14:51:51.284142 5734 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260927 14:51:51.284826 5707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:51:51.267990 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36664 (local address 127.0.98.62:37911)
0927 14:51:51.268114 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0927 14:51:51.268120 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 14:51:51.268267 (+ 147us) server_negotiation.cc:407] Connection header received
0927 14:51:51.268428 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:51:51.268431 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:51:51.268487 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:51:51.268605 (+ 118us) server_negotiation.cc:226] Negotiated authn=JWT
0927 14:51:51.268934 (+ 329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:51:51.269781 (+ 847us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:51:51.270633 (+ 852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:51:51.270968 (+ 335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:51:51.271070 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:51:51.271349 (+ 279us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0927 14:51:51.284413 (+ 13064us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40103/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0927 14:51:51.284686 (+ 273us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:36664: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40103/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":53}
W20260927 14:51:51.284894 5707 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:36664: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40103/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20260927 14:51:51.284926 5733 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 14:51:51.267880 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37911 (local address 127.0.0.1:36664)
0927 14:51:51.268173 (+ 293us) negotiation.cc:107] Waiting for socket to connect
0927 14:51:51.268185 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 14:51:51.268279 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:51:51.268686 (+ 407us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:51:51.268689 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:51:51.268697 (+ 8us) client_negotiation.cc:189] Negotiated authn=JWT
0927 14:51:51.268813 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:51:51.268818 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:51:51.269891 (+ 1073us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:51:51.269894 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:51:51.270520 (+ 626us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:51:51.270529 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:51:51.271073 (+ 544us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:51:51.271090 (+ 17us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:51:51.271152 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:51:51.271173 (+ 21us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0927 14:51:51.284622 (+ 13449us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40103/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0927 14:51:51.284758 (+ 136us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.98.62:37911: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40103/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":110,"threads_started":1}
I20260927 14:51:51.285607 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6936m5/build/debug/bin/kudu with pid 5637
2026-09-27T14:51:51Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (306 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260927 14:51:51.324748 392 mini_oidc.cc:152] Starting JWKS server
I20260927 14:51:51.325717 392 webserver.cc:460] Webserver started at https://127.0.0.1:41405/ using document root <none> and password file <none>
I20260927 14:51:51.325886 392 mini_oidc.cc:189] Starting OIDC Discovery server
I20260927 14:51:51.326146 392 webserver.cc:460] Webserver started at http://127.0.0.1:37091/ using document root <none> and password file <none>
2026-09-27T14:51:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T14:51:51Z Disabled control of system clock
I20260927 14:51:51.335253 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6936m5/build/debug/bin/kudu
/tmp/dist-test-task6936m5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790520684602251-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790520684602251-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790520684602251-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790520684602251-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36131
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:38687
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:36131
--enable_jwt_token_auth=true
--jwks_url=https://localhost:41405/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260927 14:51:51.442397 5740 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 14:51:51.442651 5740 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 14:51:51.442691 5740 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 14:51:51.446578 5740 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 14:51:51.446632 5740 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 14:51:51.446647 5740 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 14:51:51.446664 5740 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 14:51:51.446679 5740 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260927 14:51:51.446694 5740 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260927 14:51:51.446707 5740 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:41405/jwks.json
I20260927 14:51:51.451478 5740 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38687
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790520684602251-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790520684602251-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:36131
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36131
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:41405/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790520684602251-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5740
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790520684602251-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:15:17 UTC on ccycloud.quasar-ledxfe.root.comops.site
build id 4676134
I20260927 14:51:51.452601 5740 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 14:51:51.453955 5740 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 14:51:51.462359 5748 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 14:51:51.462540 5740 server_base.cc:1034] running on GCE node
W20260927 14:51:51.462405 5746 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 14:51:51.462359 5745 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 14:51:51.463393 5740 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 14:51:51.464617 5740 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 14:51:51.465838 5740 hybrid_clock.cc:648] HybridClock initialized: now 1790520711465828 us; error 31 us; skew 500 ppm
I20260927 14:51:51.467976 5740 webserver.cc:460] Webserver started at http://127.0.98.62:34145/ using document root <none> and password file <none>
I20260927 14:51:51.468591 5740 fs_manager.cc:359] Metadata directory not provided
I20260927 14:51:51.468694 5740 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 14:51:51.468937 5740 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 14:51:51.471100 5740 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790520684602251-392-0/minicluster-data/master-0/data/instance:
uuid: "6157c47063064ce8b614c0dd6b49e77f"
format_stamp: "Formatted at 2026-09-27 14:51:51 on dist-test-slave-xk1g"
server_key: "762ef455d91e18be88aedcadb4c55ac3"
server_key_iv: "ee37a7c57c58dfa2f654ebe3cffdec88"
server_key_version: "encryptionkey@0"
I20260927 14:51:51.471700 5740 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790520684602251-392-0/minicluster-data/master-0/wal/instance:
uuid: "6157c47063064ce8b614c0dd6b49e77f"
format_stamp: "Formatted at 2026-09-27 14:51:51 on dist-test-slave-xk1g"
server_key: "762ef455d91e18be88aedcadb4c55ac3"
server_key_iv: "ee37a7c57c58dfa2f654ebe3cffdec88"
server_key_version: "encryptionkey@0"
I20260927 14:51:51.475474 5740 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260927 14:51:51.477916 5754 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 14:51:51.479122 5740 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 14:51:51.479275 5740 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790520684602251-392-0/minicluster-data/master-0/data,/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790520684602251-392-0/minicluster-data/master-0/wal
uuid: "6157c47063064ce8b614c0dd6b49e77f"
format_stamp: "Formatted at 2026-09-27 14:51:51 on dist-test-slave-xk1g"
server_key: "762ef455d91e18be88aedcadb4c55ac3"
server_key_iv: "ee37a7c57c58dfa2f654ebe3cffdec88"
server_key_version: "encryptionkey@0"
I20260927 14:51:51.479413 5740 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790520684602251-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790520684602251-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790520684602251-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 14:51:51.494880 5740 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 14:51:51.495581 5740 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 14:51:51.495769 5740 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 14:51:51.502406 5740 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:36131
I20260927 14:51:51.502434 5806 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:36131 every 8 connection(s)
I20260927 14:51:51.503310 5740 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790520684602251-392-0/minicluster-data/master-0/data/info.pb
I20260927 14:51:51.506445 5807 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 14:51:51.510838 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6936m5/build/debug/bin/kudu as pid 5740
I20260927 14:51:51.510970 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790520684602251-392-0/minicluster-data/master-0/wal/instance
I20260927 14:51:51.511284 392 external_mini_cluster.cc:1428] Setting key 5c04de7ff3343294a284f6879eef70e9
I20260927 14:51:51.511786 5807 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6157c47063064ce8b614c0dd6b49e77f: Bootstrap starting.
I20260927 14:51:51.514187 5807 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6157c47063064ce8b614c0dd6b49e77f: Neither blocks nor log segments found. Creating new log.
I20260927 14:51:51.514951 5807 log.cc:824] T 00000000000000000000000000000000 P 6157c47063064ce8b614c0dd6b49e77f: Log is configured to *not* fsync() on all Append() calls
I20260927 14:51:51.516693 5807 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6157c47063064ce8b614c0dd6b49e77f: No bootstrap required, opened a new log
I20260927 14:51:51.520046 5807 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6157c47063064ce8b614c0dd6b49e77f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6157c47063064ce8b614c0dd6b49e77f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36131 } }
I20260927 14:51:51.520283 5807 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6157c47063064ce8b614c0dd6b49e77f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 14:51:51.520355 5807 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6157c47063064ce8b614c0dd6b49e77f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6157c47063064ce8b614c0dd6b49e77f, State: Initialized, Role: FOLLOWER
I20260927 14:51:51.520677 5810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:51:51.512647 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53814 (local address 127.0.98.62:36131)
0927 14:51:51.513347 (+ 700us) server_negotiation.cc:206] Beginning negotiation
0927 14:51:51.513363 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0927 14:51:51.513398 (+ 35us) server_negotiation.cc:407] Connection header received
0927 14:51:51.514039 (+ 641us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:51:51.514056 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:51:51.514497 (+ 441us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:51:51.514844 (+ 347us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:51:51.515263 (+ 419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:51:51.516220 (+ 957us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:51:51.517289 (+ 1069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:51:51.517777 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:51:51.517969 (+ 192us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:51:51.518869 (+ 900us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:51:51.518875 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:51:51.518887 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 14:51:51.519263 (+ 376us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:51:51.519320 (+ 57us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 14:51:51.519340 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:51:51.519454 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0927 14:51:51.519577 (+ 123us) server_negotiation.cc:299] Negotiation successful
0927 14:51:51.519847 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":285,"thread_start_us":137,"threads_started":1}
I20260927 14:51:51.520848 5807 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6157c47063064ce8b614c0dd6b49e77f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6157c47063064ce8b614c0dd6b49e77f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36131 } }
I20260927 14:51:51.521003 5807 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6157c47063064ce8b614c0dd6b49e77f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 14:51:51.521087 5807 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6157c47063064ce8b614c0dd6b49e77f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 14:51:51.521181 5807 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6157c47063064ce8b614c0dd6b49e77f [term 0 FOLLOWER]: Advancing to term 1
I20260927 14:51:51.522106 5807 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6157c47063064ce8b614c0dd6b49e77f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6157c47063064ce8b614c0dd6b49e77f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36131 } }
I20260927 14:51:51.522454 5807 leader_election.cc:302] T 00000000000000000000000000000000 P 6157c47063064ce8b614c0dd6b49e77f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6157c47063064ce8b614c0dd6b49e77f; no voters:
I20260927 14:51:51.522775 5807 leader_election.cc:288] T 00000000000000000000000000000000 P 6157c47063064ce8b614c0dd6b49e77f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 14:51:51.522833 5812 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6157c47063064ce8b614c0dd6b49e77f [term 1 FOLLOWER]: Leader election won for term 1
I20260927 14:51:51.523037 5812 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6157c47063064ce8b614c0dd6b49e77f [term 1 LEADER]: Becoming Leader. State: Replica: 6157c47063064ce8b614c0dd6b49e77f, State: Running, Role: LEADER
I20260927 14:51:51.523350 5812 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6157c47063064ce8b614c0dd6b49e77f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6157c47063064ce8b614c0dd6b49e77f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36131 } }
I20260927 14:51:51.523583 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 14:51:51.524008 5807 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6157c47063064ce8b614c0dd6b49e77f [sys.catalog]: configured and running, proceeding with master startup.
I20260927 14:51:51.525133 5814 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6157c47063064ce8b614c0dd6b49e77f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6157c47063064ce8b614c0dd6b49e77f. Latest consensus state: current_term: 1 leader_uuid: "6157c47063064ce8b614c0dd6b49e77f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6157c47063064ce8b614c0dd6b49e77f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36131 } } }
I20260927 14:51:51.525547 5814 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6157c47063064ce8b614c0dd6b49e77f [sys.catalog]: This master's current role is: LEADER
I20260927 14:51:51.525424 5813 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6157c47063064ce8b614c0dd6b49e77f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6157c47063064ce8b614c0dd6b49e77f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6157c47063064ce8b614c0dd6b49e77f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36131 } } }
I20260927 14:51:51.525900 5813 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6157c47063064ce8b614c0dd6b49e77f [sys.catalog]: This master's current role is: LEADER
I20260927 14:51:51.525980 5824 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 14:51:51.529454 5824 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 14:51:51.534456 5824 catalog_manager.cc:1347] Generated new cluster ID: 22662e42e2484eb7939be344ed3c7f2a
I20260927 14:51:51.534523 5824 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 14:51:51.544314 5824 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 14:51:51.545243 5824 catalog_manager.cc:1504] Loading token signing keys...
I20260927 14:51:51.550469 5824 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6157c47063064ce8b614c0dd6b49e77f: Generated new TSK 0
I20260927 14:51:51.551172 5824 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 14:51:51.573300 5810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:51:51.568217 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53818 (local address 127.0.98.62:36131)
0927 14:51:51.568359 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0927 14:51:51.568363 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 14:51:51.568573 (+ 210us) server_negotiation.cc:407] Connection header received
0927 14:51:51.568731 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:51:51.568733 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:51:51.568780 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:51:51.568869 (+ 89us) server_negotiation.cc:226] Negotiated authn=JWT
0927 14:51:51.569180 (+ 311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:51:51.569696 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:51:51.570944 (+ 1248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:51:51.571180 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:51:51.571273 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:51:51.571620 (+ 347us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0927 14:51:51.572872 (+ 1252us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41405/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0927 14:51:51.573159 (+ 287us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:53818: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41405/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":56}
W20260927 14:51:51.573378 5836 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 14:51:51.568157 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36131 (local address 127.0.0.1:53818)
0927 14:51:51.568478 (+ 321us) negotiation.cc:107] Waiting for socket to connect
0927 14:51:51.568493 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 14:51:51.568610 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:51:51.568899 (+ 289us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:51:51.568902 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:51:51.568911 (+ 9us) client_negotiation.cc:189] Negotiated authn=JWT
0927 14:51:51.569035 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:51:51.569041 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:51:51.569804 (+ 763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:51:51.569808 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:51:51.570797 (+ 989us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:51:51.570807 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:51:51.571288 (+ 481us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:51:51.571314 (+ 26us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:51:51.571405 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:51:51.571434 (+ 29us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0927 14:51:51.573030 (+ 1596us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41405/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0927 14:51:51.573191 (+ 161us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.98.62:36131: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41405/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":238,"thread_start_us":136,"threads_started":1}
W20260927 14:51:51.573765 5810 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:53818: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41405/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
I20260927 14:51:51.573879 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6936m5/build/debug/bin/kudu with pid 5740
E20260927 14:51:51.584676 5837 webserver.cc:533] Webserver: sslize failed:
E20260927 14:51:51.584784 5837 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
2026-09-27T14:51:51Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (287 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-09-27T14:51:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T14:51:51Z Disabled control of system clock
I20260927 14:51:51.620435 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6936m5/build/debug/bin/kudu
/tmp/dist-test-task6936m5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790520684602251-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790520684602251-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790520684602251-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790520684602251-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35347
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:35561
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:35347
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790520684602251-392-0/testchainca.cert with env {}
W20260927 14:51:51.728642 5841 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 14:51:51.728935 5841 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 14:51:51.728974 5841 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 14:51:51.733186 5841 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 14:51:51.733247 5841 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 14:51:51.733263 5841 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 14:51:51.733279 5841 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 14:51:51.733294 5841 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260927 14:51:51.733310 5841 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260927 14:51:51.733325 5841 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260927 14:51:51.738631 5841 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35561
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790520684602251-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790520684602251-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:35347
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35347
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790520684602251-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790520684602251-392-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5841
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790520684602251-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:15:17 UTC on ccycloud.quasar-ledxfe.root.comops.site
build id 4676134
I20260927 14:51:51.739629 5841 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 14:51:51.740571 5841 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 14:51:51.748643 5847 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 14:51:51.748684 5846 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 14:51:51.748639 5849 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 14:51:51.748818 5841 server_base.cc:1034] running on GCE node
I20260927 14:51:51.749588 5841 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 14:51:51.750582 5841 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 14:51:51.751747 5841 hybrid_clock.cc:648] HybridClock initialized: now 1790520711751721 us; error 49 us; skew 500 ppm
I20260927 14:51:51.753602 5841 webserver.cc:460] Webserver started at http://127.0.98.62:35373/ using document root <none> and password file <none>
I20260927 14:51:51.754133 5841 fs_manager.cc:359] Metadata directory not provided
I20260927 14:51:51.754194 5841 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 14:51:51.754366 5841 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 14:51:51.756315 5841 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790520684602251-392-0/minicluster-data/master-0/data/instance:
uuid: "562772d43a9a41318a6dfd7abc0db9d7"
format_stamp: "Formatted at 2026-09-27 14:51:51 on dist-test-slave-xk1g"
server_key: "ddcb5b40b1264f5fe798d8bba4194fdb"
server_key_iv: "be27075a6c2261006596465260b6ce67"
server_key_version: "encryptionkey@0"
I20260927 14:51:51.756825 5841 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790520684602251-392-0/minicluster-data/master-0/wal/instance:
uuid: "562772d43a9a41318a6dfd7abc0db9d7"
format_stamp: "Formatted at 2026-09-27 14:51:51 on dist-test-slave-xk1g"
server_key: "ddcb5b40b1264f5fe798d8bba4194fdb"
server_key_iv: "be27075a6c2261006596465260b6ce67"
server_key_version: "encryptionkey@0"
I20260927 14:51:51.760413 5841 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260927 14:51:51.762758 5855 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 14:51:51.764010 5841 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s
I20260927 14:51:51.764144 5841 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790520684602251-392-0/minicluster-data/master-0/data,/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790520684602251-392-0/minicluster-data/master-0/wal
uuid: "562772d43a9a41318a6dfd7abc0db9d7"
format_stamp: "Formatted at 2026-09-27 14:51:51 on dist-test-slave-xk1g"
server_key: "ddcb5b40b1264f5fe798d8bba4194fdb"
server_key_iv: "be27075a6c2261006596465260b6ce67"
server_key_version: "encryptionkey@0"
I20260927 14:51:51.764254 5841 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790520684602251-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790520684602251-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790520684602251-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 14:51:51.792514 5841 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 14:51:51.793272 5841 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 14:51:51.793462 5841 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 14:51:51.800506 5841 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:35347
I20260927 14:51:51.800534 5907 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:35347 every 8 connection(s)
I20260927 14:51:51.801549 5841 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790520684602251-392-0/minicluster-data/master-0/data/info.pb
I20260927 14:51:51.804661 5908 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 14:51:51.805900 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6936m5/build/debug/bin/kudu as pid 5841
I20260927 14:51:51.806033 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790520684602251-392-0/minicluster-data/master-0/wal/instance
I20260927 14:51:51.806354 392 external_mini_cluster.cc:1428] Setting key f7e1716a9b0c6575cdb2f2918e3365f1
I20260927 14:51:51.810859 5908 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 562772d43a9a41318a6dfd7abc0db9d7: Bootstrap starting.
I20260927 14:51:51.813118 5908 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 562772d43a9a41318a6dfd7abc0db9d7: Neither blocks nor log segments found. Creating new log.
I20260927 14:51:51.813964 5908 log.cc:824] T 00000000000000000000000000000000 P 562772d43a9a41318a6dfd7abc0db9d7: Log is configured to *not* fsync() on all Append() calls
I20260927 14:51:51.815843 5911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:51:51.807883 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46862 (local address 127.0.98.62:35347)
0927 14:51:51.808541 (+ 658us) server_negotiation.cc:206] Beginning negotiation
0927 14:51:51.808565 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0927 14:51:51.808601 (+ 36us) server_negotiation.cc:407] Connection header received
0927 14:51:51.809316 (+ 715us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:51:51.809335 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:51:51.809725 (+ 390us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:51:51.810067 (+ 342us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:51:51.810516 (+ 449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:51:51.811564 (+ 1048us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:51:51.812420 (+ 856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:51:51.812932 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:51:51.813143 (+ 211us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:51:51.814019 (+ 876us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:51:51.814024 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:51:51.814036 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 14:51:51.814537 (+ 501us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:51:51.814606 (+ 69us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 14:51:51.814635 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:51:51.814736 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0927 14:51:51.814874 (+ 138us) server_negotiation.cc:299] Negotiation successful
0927 14:51:51.815156 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":298,"thread_start_us":127,"threads_started":1}
I20260927 14:51:51.816221 5908 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 562772d43a9a41318a6dfd7abc0db9d7: No bootstrap required, opened a new log
I20260927 14:51:51.818807 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 14:51:51.820170 5908 raft_consensus.cc:348] T 00000000000000000000000000000000 P 562772d43a9a41318a6dfd7abc0db9d7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "562772d43a9a41318a6dfd7abc0db9d7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35347 } }
I20260927 14:51:51.820466 5908 raft_consensus.cc:374] T 00000000000000000000000000000000 P 562772d43a9a41318a6dfd7abc0db9d7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 14:51:51.820554 5908 raft_consensus.cc:729] T 00000000000000000000000000000000 P 562772d43a9a41318a6dfd7abc0db9d7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 562772d43a9a41318a6dfd7abc0db9d7, State: Initialized, Role: FOLLOWER
I20260927 14:51:51.821000 5908 consensus_queue.cc:260] T 00000000000000000000000000000000 P 562772d43a9a41318a6dfd7abc0db9d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "562772d43a9a41318a6dfd7abc0db9d7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35347 } }
I20260927 14:51:51.821197 5908 raft_consensus.cc:388] T 00000000000000000000000000000000 P 562772d43a9a41318a6dfd7abc0db9d7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 14:51:51.821262 5908 raft_consensus.cc:482] T 00000000000000000000000000000000 P 562772d43a9a41318a6dfd7abc0db9d7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 14:51:51.821355 5908 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 562772d43a9a41318a6dfd7abc0db9d7 [term 0 FOLLOWER]: Advancing to term 1
I20260927 14:51:51.822255 5908 raft_consensus.cc:504] T 00000000000000000000000000000000 P 562772d43a9a41318a6dfd7abc0db9d7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "562772d43a9a41318a6dfd7abc0db9d7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35347 } }
I20260927 14:51:51.822590 5908 leader_election.cc:302] T 00000000000000000000000000000000 P 562772d43a9a41318a6dfd7abc0db9d7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 562772d43a9a41318a6dfd7abc0db9d7; no voters:
I20260927 14:51:51.822901 5908 leader_election.cc:288] T 00000000000000000000000000000000 P 562772d43a9a41318a6dfd7abc0db9d7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 14:51:51.823028 5913 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 562772d43a9a41318a6dfd7abc0db9d7 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 14:51:51.823268 5913 raft_consensus.cc:686] T 00000000000000000000000000000000 P 562772d43a9a41318a6dfd7abc0db9d7 [term 1 LEADER]: Becoming Leader. State: Replica: 562772d43a9a41318a6dfd7abc0db9d7, State: Running, Role: LEADER
I20260927 14:51:51.823756 5913 consensus_queue.cc:237] T 00000000000000000000000000000000 P 562772d43a9a41318a6dfd7abc0db9d7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "562772d43a9a41318a6dfd7abc0db9d7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35347 } }
I20260927 14:51:51.824200 5908 sys_catalog.cc:564] T 00000000000000000000000000000000 P 562772d43a9a41318a6dfd7abc0db9d7 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 14:51:51.825464 5915 sys_catalog.cc:455] T 00000000000000000000000000000000 P 562772d43a9a41318a6dfd7abc0db9d7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 562772d43a9a41318a6dfd7abc0db9d7. Latest consensus state: current_term: 1 leader_uuid: "562772d43a9a41318a6dfd7abc0db9d7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "562772d43a9a41318a6dfd7abc0db9d7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35347 } } }
I20260927 14:51:51.825603 5915 sys_catalog.cc:458] T 00000000000000000000000000000000 P 562772d43a9a41318a6dfd7abc0db9d7 [sys.catalog]: This master's current role is: LEADER
I20260927 14:51:51.825979 5922 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 14:51:51.825922 5914 sys_catalog.cc:455] T 00000000000000000000000000000000 P 562772d43a9a41318a6dfd7abc0db9d7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "562772d43a9a41318a6dfd7abc0db9d7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "562772d43a9a41318a6dfd7abc0db9d7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35347 } } }
I20260927 14:51:51.826104 5914 sys_catalog.cc:458] T 00000000000000000000000000000000 P 562772d43a9a41318a6dfd7abc0db9d7 [sys.catalog]: This master's current role is: LEADER
I20260927 14:51:51.829465 5922 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 14:51:51.835124 5922 catalog_manager.cc:1347] Generated new cluster ID: 867e146c8b6f4277917f783872ea8d23
I20260927 14:51:51.835196 5922 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 14:51:51.851858 5922 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 14:51:51.852843 5922 catalog_manager.cc:1504] Loading token signing keys...
I20260927 14:51:51.860147 5922 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 562772d43a9a41318a6dfd7abc0db9d7: Generated new TSK 0
I20260927 14:51:51.860870 5922 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 14:51:52.343806 5911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:51:51.876504 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46866 (local address 127.0.98.62:35347)
0927 14:51:51.876641 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0927 14:51:51.876646 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 14:51:51.876832 (+ 186us) server_negotiation.cc:407] Connection header received
0927 14:51:51.877023 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:51:51.877026 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:51:51.877141 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:51:51.877237 (+ 96us) server_negotiation.cc:226] Negotiated authn=JWT
0927 14:51:51.877502 (+ 265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:51:51.878022 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:51:51.879260 (+ 1238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:51:51.879482 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:51:51.879561 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:51:51.879920 (+ 359us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0927 14:51:52.343379 (+463459us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0927 14:51:52.343673 (+ 294us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:46866: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":54}
W20260927 14:51:52.343891 5911 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:46866: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260927 14:51:52.344061 5937 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 14:51:51.876437 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35347 (local address 127.0.0.1:46866)
0927 14:51:51.876741 (+ 304us) negotiation.cc:107] Waiting for socket to connect
0927 14:51:51.876755 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 14:51:51.876867 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:51:51.877262 (+ 395us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:51:51.877265 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:51:51.877273 (+ 8us) client_negotiation.cc:189] Negotiated authn=JWT
0927 14:51:51.877397 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:51:51.877403 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:51:51.878126 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:51:51.878130 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:51:51.879128 (+ 998us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:51:51.879138 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:51:51.879578 (+ 440us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:51:51.879601 (+ 23us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:51:51.879696 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:51:51.879724 (+ 28us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0927 14:51:52.343627 (+463903us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0927 14:51:52.343804 (+ 177us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.98.62:35347: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":218,"thread_start_us":111,"threads_started":1}
I20260927 14:51:52.344655 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6936m5/build/debug/bin/kudu with pid 5841
2026-09-27T14:51:52Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (751 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (121 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (123 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790520684602251-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 14:51:52 dist-test-slave-xk1g krb5kdc[5949](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 14:51:52 dist-test-slave-xk1g krb5kdc[5949](info): set up 2 sockets
Sep 27 14:51:52 dist-test-slave-xk1g krb5kdc[5949](info): commencing operation
krb5kdc: starting...
W20260927 14:51:53.625357 392 mini_kdc.cc:121] Time spent starting KDC: real 1.016s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790520684602251-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790520684602251-392-0/krb5kdc/test-user.keytab.
Sep 27 14:51:53 dist-test-slave-xk1g krb5kdc[5949](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790520713, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T14:51:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T14:51:53Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790520684602251-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790520684602251-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 14:51:53.747687 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6936m5/build/debug/bin/kudu
/tmp/dist-test-task6936m5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790520684602251-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790520684602251-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790520684602251-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790520684602251-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35495
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790520684602251-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36185
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:35495
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790520684602251-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790520684602251-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790520684602251-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 14:51:53.853590 5962 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 14:51:53.853871 5962 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 14:51:53.853919 5962 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 14:51:53.857859 5962 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 14:51:53.857918 5962 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 14:51:53.857936 5962 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 14:51:53.857952 5962 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 14:51:53.857967 5962 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 14:51:53.863050 5962 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36185
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790520684602251-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790520684602251-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:35495
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790520684602251-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35495
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790520684602251-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5962
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790520684602251-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:15:17 UTC on ccycloud.quasar-ledxfe.root.comops.site
build id 4676134
I20260927 14:51:53.864037 5962 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 14:51:53.864982 5962 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 14:51:53.873181 5970 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 14:51:53.873199 5967 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 14:51:53.873347 5962 server_base.cc:1034] running on GCE node
W20260927 14:51:53.873188 5968 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 14:51:53.874064 5962 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 14:51:53.875133 5962 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 14:51:53.876304 5962 hybrid_clock.cc:648] HybridClock initialized: now 1790520713876277 us; error 39 us; skew 500 ppm
Sep 27 14:51:53 dist-test-slave-xk1g krb5kdc[5949](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790520713, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 14:51:53.882443 5962 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 14:51:53.883352 5962 webserver.cc:460] Webserver started at http://127.0.98.62:33685/ using document root <none> and password file <none>
I20260927 14:51:53.883944 5962 fs_manager.cc:359] Metadata directory not provided
I20260927 14:51:53.884006 5962 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 14:51:53.884227 5962 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 14:51:53.886452 5962 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790520684602251-392-0/minicluster-data/master-0/data/instance:
uuid: "71b05cd837e244118a75e2745a0108cb"
format_stamp: "Formatted at 2026-09-27 14:51:53 on dist-test-slave-xk1g"
server_key: "29b43d9628da77ad1c8d399e22083751"
server_key_iv: "614b754512d3553db9361fb5d5e2676e"
server_key_version: "encryptionkey@0"
I20260927 14:51:53.887303 5962 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790520684602251-392-0/minicluster-data/master-0/wal/instance:
uuid: "71b05cd837e244118a75e2745a0108cb"
format_stamp: "Formatted at 2026-09-27 14:51:53 on dist-test-slave-xk1g"
server_key: "29b43d9628da77ad1c8d399e22083751"
server_key_iv: "614b754512d3553db9361fb5d5e2676e"
server_key_version: "encryptionkey@0"
I20260927 14:51:53.892915 5962 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260927 14:51:53.895754 5977 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 14:51:53.896966 5962 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 14:51:53.897125 5962 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790520684602251-392-0/minicluster-data/master-0/data,/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790520684602251-392-0/minicluster-data/master-0/wal
uuid: "71b05cd837e244118a75e2745a0108cb"
format_stamp: "Formatted at 2026-09-27 14:51:53 on dist-test-slave-xk1g"
server_key: "29b43d9628da77ad1c8d399e22083751"
server_key_iv: "614b754512d3553db9361fb5d5e2676e"
server_key_version: "encryptionkey@0"
I20260927 14:51:53.897246 5962 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790520684602251-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790520684602251-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790520684602251-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 14:51:53.906271 5962 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 14:51:53.908974 5962 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 14:51:53.909186 5962 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 14:51:53.916190 5962 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:35495
I20260927 14:51:53.916466 6029 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:35495 every 8 connection(s)
I20260927 14:51:53.917212 5962 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790520684602251-392-0/minicluster-data/master-0/data/info.pb
I20260927 14:51:53.919826 6030 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 14:51:53.923300 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6936m5/build/debug/bin/kudu as pid 5962
I20260927 14:51:53.923453 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790520684602251-392-0/minicluster-data/master-0/wal/instance
I20260927 14:51:53.923712 392 external_mini_cluster.cc:1428] Setting key 039e17bc02f05d8736a713b408221d7b
I20260927 14:51:53.925606 6030 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 71b05cd837e244118a75e2745a0108cb: Bootstrap starting.
I20260927 14:51:53.928223 6030 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 71b05cd837e244118a75e2745a0108cb: Neither blocks nor log segments found. Creating new log.
I20260927 14:51:53.929091 6030 log.cc:824] T 00000000000000000000000000000000 P 71b05cd837e244118a75e2745a0108cb: Log is configured to *not* fsync() on all Append() calls
I20260927 14:51:53.931097 6030 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 71b05cd837e244118a75e2745a0108cb: No bootstrap required, opened a new log
Sep 27 14:51:53 dist-test-slave-xk1g krb5kdc[5949](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790520713, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 14:51:53.935015 6030 raft_consensus.cc:348] T 00000000000000000000000000000000 P 71b05cd837e244118a75e2745a0108cb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "71b05cd837e244118a75e2745a0108cb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35495 } }
I20260927 14:51:53.935286 6030 raft_consensus.cc:374] T 00000000000000000000000000000000 P 71b05cd837e244118a75e2745a0108cb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 14:51:53.935344 6030 raft_consensus.cc:729] T 00000000000000000000000000000000 P 71b05cd837e244118a75e2745a0108cb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 71b05cd837e244118a75e2745a0108cb, State: Initialized, Role: FOLLOWER
I20260927 14:51:53.935842 6030 consensus_queue.cc:260] T 00000000000000000000000000000000 P 71b05cd837e244118a75e2745a0108cb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "71b05cd837e244118a75e2745a0108cb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35495 } }
I20260927 14:51:53.935994 6030 raft_consensus.cc:388] T 00000000000000000000000000000000 P 71b05cd837e244118a75e2745a0108cb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 14:51:53.936081 6030 raft_consensus.cc:482] T 00000000000000000000000000000000 P 71b05cd837e244118a75e2745a0108cb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 14:51:53.936175 6030 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 71b05cd837e244118a75e2745a0108cb [term 0 FOLLOWER]: Advancing to term 1
I20260927 14:51:53.937009 6030 raft_consensus.cc:504] T 00000000000000000000000000000000 P 71b05cd837e244118a75e2745a0108cb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "71b05cd837e244118a75e2745a0108cb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35495 } }
I20260927 14:51:53.937367 6030 leader_election.cc:302] T 00000000000000000000000000000000 P 71b05cd837e244118a75e2745a0108cb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 71b05cd837e244118a75e2745a0108cb; no voters:
I20260927 14:51:53.937696 6030 leader_election.cc:288] T 00000000000000000000000000000000 P 71b05cd837e244118a75e2745a0108cb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 14:51:53.937790 6035 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 71b05cd837e244118a75e2745a0108cb [term 1 FOLLOWER]: Leader election won for term 1
I20260927 14:51:53.937996 6035 raft_consensus.cc:686] T 00000000000000000000000000000000 P 71b05cd837e244118a75e2745a0108cb [term 1 LEADER]: Becoming Leader. State: Replica: 71b05cd837e244118a75e2745a0108cb, State: Running, Role: LEADER
I20260927 14:51:53.938330 6035 consensus_queue.cc:237] T 00000000000000000000000000000000 P 71b05cd837e244118a75e2745a0108cb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "71b05cd837e244118a75e2745a0108cb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35495 } }
I20260927 14:51:53.938884 6033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:51:53.925165 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59396 (local address 127.0.98.62:35495)
0927 14:51:53.925591 (+ 426us) server_negotiation.cc:206] Beginning negotiation
0927 14:51:53.925611 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0927 14:51:53.925654 (+ 43us) server_negotiation.cc:407] Connection header received
0927 14:51:53.926434 (+ 780us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:51:53.926455 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:51:53.926899 (+ 444us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:51:53.927247 (+ 348us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:51:53.927888 (+ 641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:51:53.929396 (+ 1508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:51:53.930351 (+ 955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:51:53.931019 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:51:53.931181 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:51:53.933679 (+ 2498us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:51:53.933688 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:51:53.933704 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 14:51:53.933733 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:51:53.935742 (+ 2009us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:51:53.936226 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:51:53.936234 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:51:53.936244 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:51:53.936404 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:51:53.936724 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:51:53.936729 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:51:53.936731 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:51:53.937243 (+ 512us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:51:53.937445 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0927 14:51:53.937704 (+ 259us) server_negotiation.cc:299] Negotiation successful
0927 14:51:53.938035 (+ 331us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":287,"thread_start_us":116,"threads_started":1}
I20260927 14:51:53.938993 6030 sys_catalog.cc:564] T 00000000000000000000000000000000 P 71b05cd837e244118a75e2745a0108cb [sys.catalog]: configured and running, proceeding with master startup.
I20260927 14:51:53.939936 6036 sys_catalog.cc:455] T 00000000000000000000000000000000 P 71b05cd837e244118a75e2745a0108cb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "71b05cd837e244118a75e2745a0108cb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "71b05cd837e244118a75e2745a0108cb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35495 } } }
I20260927 14:51:53.940069 6036 sys_catalog.cc:458] T 00000000000000000000000000000000 P 71b05cd837e244118a75e2745a0108cb [sys.catalog]: This master's current role is: LEADER
I20260927 14:51:53.940292 6036 sys_catalog.cc:455] T 00000000000000000000000000000000 P 71b05cd837e244118a75e2745a0108cb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 71b05cd837e244118a75e2745a0108cb. Latest consensus state: current_term: 1 leader_uuid: "71b05cd837e244118a75e2745a0108cb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "71b05cd837e244118a75e2745a0108cb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35495 } } }
I20260927 14:51:53.940366 6036 sys_catalog.cc:458] T 00000000000000000000000000000000 P 71b05cd837e244118a75e2745a0108cb [sys.catalog]: This master's current role is: LEADER
I20260927 14:51:53.941390 6044 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 14:51:53.943918 6044 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 14:51:53.949712 6044 catalog_manager.cc:1347] Generated new cluster ID: a118fa30b78a40949731e4ff5c83dad7
I20260927 14:51:53.949793 6044 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 14:51:53.967228 6044 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 14:51:53.968542 6044 catalog_manager.cc:1504] Loading token signing keys...
I20260927 14:51:53.974076 6044 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 71b05cd837e244118a75e2745a0108cb: Generated new TSK 0
I20260927 14:51:53.974742 6044 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790520684602251-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790520684602251-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260927 14:51:54.016166 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6936m5/build/debug/bin/kudu
/tmp/dist-test-task6936m5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790520684602251-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790520684602251-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790520684602251-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790520684602251-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35495
--keytab_file=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790520684602251-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36185
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790520684602251-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790520684602251-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790520684602251-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 14:51:54.124296 6056 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 14:51:54.124545 6056 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 14:51:54.124588 6056 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 14:51:54.128572 6056 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 14:51:54.128625 6056 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 14:51:54.128682 6056 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 14:51:54.133888 6056 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36185
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790520684602251-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790520684602251-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790520684602251-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790520684602251-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35495
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6056
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790520684602251-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:15:17 UTC on ccycloud.quasar-ledxfe.root.comops.site
build id 4676134
I20260927 14:51:54.134917 6056 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 14:51:54.136018 6056 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 14:51:54.145200 6061 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 14:51:54.145206 6064 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 14:51:54.145318 6062 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 14:51:54.145256 6056 server_base.cc:1034] running on GCE node
I20260927 14:51:54.145812 6056 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 14:51:54.146449 6056 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 14:51:54.147629 6056 hybrid_clock.cc:648] HybridClock initialized: now 1790520714147620 us; error 31 us; skew 500 ppm
Sep 27 14:51:54 dist-test-slave-xk1g krb5kdc[5949](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790520714, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 14:51:54.150309 6056 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 14:51:54.151103 6056 webserver.cc:460] Webserver started at http://127.0.98.1:36221/ using document root <none> and password file <none>
I20260927 14:51:54.151641 6056 fs_manager.cc:359] Metadata directory not provided
I20260927 14:51:54.151731 6056 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 14:51:54.151973 6056 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 14:51:54.153970 6056 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790520684602251-392-0/minicluster-data/ts-0/data/instance:
uuid: "479c01219be94832acce604ab49a0824"
format_stamp: "Formatted at 2026-09-27 14:51:54 on dist-test-slave-xk1g"
server_key: "d3a007ed4240a14264efa80cb7c71d41"
server_key_iv: "b2f672cc9d425bcec37d4af3b6f45b35"
server_key_version: "encryptionkey@0"
I20260927 14:51:54.154513 6056 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790520684602251-392-0/minicluster-data/ts-0/wal/instance:
uuid: "479c01219be94832acce604ab49a0824"
format_stamp: "Formatted at 2026-09-27 14:51:54 on dist-test-slave-xk1g"
server_key: "d3a007ed4240a14264efa80cb7c71d41"
server_key_iv: "b2f672cc9d425bcec37d4af3b6f45b35"
server_key_version: "encryptionkey@0"
I20260927 14:51:54.158039 6056 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 14:51:54.160370 6071 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 14:51:54.161800 6056 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 14:51:54.161931 6056 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790520684602251-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790520684602251-392-0/minicluster-data/ts-0/wal
uuid: "479c01219be94832acce604ab49a0824"
format_stamp: "Formatted at 2026-09-27 14:51:54 on dist-test-slave-xk1g"
server_key: "d3a007ed4240a14264efa80cb7c71d41"
server_key_iv: "b2f672cc9d425bcec37d4af3b6f45b35"
server_key_version: "encryptionkey@0"
I20260927 14:51:54.162032 6056 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790520684602251-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790520684602251-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790520684602251-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 14:51:54.212980 6056 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 14:51:54.215929 6056 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 14:51:54.216145 6056 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 14:51:54.216739 6056 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 14:51:54.217638 6056 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 14:51:54.217686 6056 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 14:51:54.217756 6056 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 14:51:54.217787 6056 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 14:51:54.226742 6056 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:45245
I20260927 14:51:54.226769 6184 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:45245 every 8 connection(s)
I20260927 14:51:54.227620 6056 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790520684602251-392-0/minicluster-data/ts-0/data/info.pb
I20260927 14:51:54.232537 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6936m5/build/debug/bin/kudu as pid 6056
I20260927 14:51:54.232667 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790520684602251-392-0/minicluster-data/ts-0/wal/instance
I20260927 14:51:54.232929 392 external_mini_cluster.cc:1428] Setting key f98a2dc7686a8b684ec582269ded376b
Sep 27 14:51:54 dist-test-slave-xk1g krb5kdc[5949](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790520714, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260927 14:51:54.242731 6033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:51:54.229440 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:33591 (local address 127.0.98.62:35495)
0927 14:51:54.229593 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0927 14:51:54.229598 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 14:51:54.230483 (+ 885us) server_negotiation.cc:407] Connection header received
0927 14:51:54.231848 (+ 1365us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:51:54.231852 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:51:54.231906 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:51:54.231998 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:51:54.233275 (+ 1277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:51:54.233957 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:51:54.235371 (+ 1414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:51:54.235786 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:51:54.235919 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:51:54.238807 (+ 2888us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:51:54.238810 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:51:54.238812 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 14:51:54.238835 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:51:54.240580 (+ 1745us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:51:54.241146 (+ 566us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:51:54.241151 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:51:54.241153 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:51:54.241234 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:51:54.241583 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:51:54.241586 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:51:54.241588 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:51:54.241770 (+ 182us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:51:54.241894 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0927 14:51:54.242442 (+ 548us) server_negotiation.cc:299] Negotiation successful
0927 14:51:54.242604 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260927 14:51:54.243539 6187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:51:54.229887 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35495 (local address 127.0.98.1:33591)
0927 14:51:54.230329 (+ 442us) negotiation.cc:107] Waiting for socket to connect
0927 14:51:54.230371 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0927 14:51:54.231624 (+ 1253us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:51:54.232163 (+ 539us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:51:54.232170 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:51:54.232509 (+ 339us) client_negotiation.cc:189] Negotiated authn=SASL
0927 14:51:54.233130 (+ 621us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:51:54.233148 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:51:54.234091 (+ 943us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:51:54.234094 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:51:54.235216 (+ 1122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:51:54.235226 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:51:54.235958 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:51:54.235963 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:51:54.236205 (+ 242us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:51:54.236861 (+ 656us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 14:51:54.236890 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0927 14:51:54.238634 (+ 1744us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 14:51:54.240699 (+ 2065us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 14:51:54.240705 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 14:51:54.240721 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0927 14:51:54.241033 (+ 312us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 14:51:54.241325 (+ 292us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 14:51:54.241329 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 14:51:54.241331 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 14:51:54.241482 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 14:51:54.241893 (+ 411us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 14:51:54.241901 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 14:51:54.242152 (+ 251us) client_negotiation.cc:769] Sending connection context
0927 14:51:54.242355 (+ 203us) client_negotiation.cc:240] Negotiation successful
0927 14:51:54.242618 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":257,"thread_start_us":112,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260927 14:51:54.244911 6185 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35495
I20260927 14:51:54.245218 6185 heartbeater.cc:464] Registering TS with master...
I20260927 14:51:54.245800 6185 heartbeater.cc:511] Master 127.0.98.62:35495 requested a full tablet report, sending...
I20260927 14:51:54.247363 5994 ts_manager.cc:194] Registered new tserver with Master: 479c01219be94832acce604ab49a0824 (127.0.98.1:45245)
I20260927 14:51:54.248555 5994 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:33591
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790520684602251-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790520684602251-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260927 14:51:54.260557 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6936m5/build/debug/bin/kudu
/tmp/dist-test-task6936m5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790520684602251-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790520684602251-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790520684602251-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790520684602251-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35495
--keytab_file=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790520684602251-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36185
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790520684602251-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790520684602251-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790520684602251-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 14:51:54.368363 6190 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 14:51:54.368670 6190 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 14:51:54.368748 6190 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 14:51:54.372818 6190 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 14:51:54.372916 6190 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 14:51:54.373014 6190 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 14:51:54.378201 6190 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36185
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790520684602251-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790520684602251-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790520684602251-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790520684602251-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35495
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6190
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790520684602251-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:15:17 UTC on ccycloud.quasar-ledxfe.root.comops.site
build id 4676134
I20260927 14:51:54.379254 6190 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 14:51:54.380266 6190 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 14:51:54.389436 6198 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 14:51:54.389479 6196 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 14:51:54.389509 6195 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 14:51:54.389614 6190 server_base.cc:1034] running on GCE node
I20260927 14:51:54.390168 6190 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 14:51:54.390765 6190 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 14:51:54.391947 6190 hybrid_clock.cc:648] HybridClock initialized: now 1790520714391915 us; error 52 us; skew 500 ppm
Sep 27 14:51:54 dist-test-slave-xk1g krb5kdc[5949](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790520714, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 14:51:54.394572 6190 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 14:51:54.395462 6190 webserver.cc:460] Webserver started at http://127.0.98.2:39163/ using document root <none> and password file <none>
I20260927 14:51:54.396013 6190 fs_manager.cc:359] Metadata directory not provided
I20260927 14:51:54.396102 6190 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 14:51:54.396327 6190 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 14:51:54.398388 6190 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790520684602251-392-0/minicluster-data/ts-1/data/instance:
uuid: "f014825cff914e5aab4ebf3cdf8df37d"
format_stamp: "Formatted at 2026-09-27 14:51:54 on dist-test-slave-xk1g"
server_key: "caf1435da3719b1b7d912bef7c4c36c1"
server_key_iv: "f08769645c649e40900326f25d171fbf"
server_key_version: "encryptionkey@0"
I20260927 14:51:54.398957 6190 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790520684602251-392-0/minicluster-data/ts-1/wal/instance:
uuid: "f014825cff914e5aab4ebf3cdf8df37d"
format_stamp: "Formatted at 2026-09-27 14:51:54 on dist-test-slave-xk1g"
server_key: "caf1435da3719b1b7d912bef7c4c36c1"
server_key_iv: "f08769645c649e40900326f25d171fbf"
server_key_version: "encryptionkey@0"
I20260927 14:51:54.402702 6190 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260927 14:51:54.405171 6205 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 14:51:54.406441 6190 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 14:51:54.406584 6190 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790520684602251-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790520684602251-392-0/minicluster-data/ts-1/wal
uuid: "f014825cff914e5aab4ebf3cdf8df37d"
format_stamp: "Formatted at 2026-09-27 14:51:54 on dist-test-slave-xk1g"
server_key: "caf1435da3719b1b7d912bef7c4c36c1"
server_key_iv: "f08769645c649e40900326f25d171fbf"
server_key_version: "encryptionkey@0"
I20260927 14:51:54.406713 6190 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790520684602251-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790520684602251-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790520684602251-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 14:51:54.423977 6190 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 14:51:54.427235 6190 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 14:51:54.427449 6190 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 14:51:54.428061 6190 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 14:51:54.428892 6190 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 14:51:54.428965 6190 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 14:51:54.429031 6190 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 14:51:54.429102 6190 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 14:51:54.438301 6190 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:40739
I20260927 14:51:54.438375 6318 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:40739 every 8 connection(s)
I20260927 14:51:54.439291 6190 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790520684602251-392-0/minicluster-data/ts-1/data/info.pb
I20260927 14:51:54.446609 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6936m5/build/debug/bin/kudu as pid 6190
I20260927 14:51:54.446730 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790520684602251-392-0/minicluster-data/ts-1/wal/instance
I20260927 14:51:54.447019 392 external_mini_cluster.cc:1428] Setting key e0db6977895bb13157bb01c556661ceb
Sep 27 14:51:54 dist-test-slave-xk1g krb5kdc[5949](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790520714, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 14:51:54.453438 6033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:51:54.441299 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:50357 (local address 127.0.98.62:35495)
0927 14:51:54.441467 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0927 14:51:54.441473 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 14:51:54.442399 (+ 926us) server_negotiation.cc:407] Connection header received
0927 14:51:54.443759 (+ 1360us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:51:54.443762 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:51:54.443809 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:51:54.443887 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:51:54.445046 (+ 1159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:51:54.445589 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:51:54.446722 (+ 1133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:51:54.447040 (+ 318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:51:54.447154 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:51:54.449786 (+ 2632us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:51:54.449793 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:51:54.449799 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 14:51:54.449833 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:51:54.451363 (+ 1530us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:51:54.451876 (+ 513us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:51:54.451882 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:51:54.451884 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:51:54.451964 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:51:54.452289 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:51:54.452294 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:51:54.452295 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:51:54.452495 (+ 200us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:51:54.452603 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0927 14:51:54.453134 (+ 531us) server_negotiation.cc:299] Negotiation successful
0927 14:51:54.453258 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260927 14:51:54.454214 6321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:51:54.441754 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35495 (local address 127.0.98.2:50357)
0927 14:51:54.442236 (+ 482us) negotiation.cc:107] Waiting for socket to connect
0927 14:51:54.442283 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0927 14:51:54.443561 (+ 1278us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:51:54.444080 (+ 519us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:51:54.444087 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:51:54.444433 (+ 346us) client_negotiation.cc:189] Negotiated authn=SASL
0927 14:51:54.444906 (+ 473us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:51:54.444919 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:51:54.445701 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:51:54.445704 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:51:54.446596 (+ 892us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:51:54.446605 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:51:54.447220 (+ 615us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:51:54.447223 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:51:54.447366 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:51:54.447960 (+ 594us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 14:51:54.447979 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0927 14:51:54.449630 (+ 1651us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 14:51:54.451482 (+ 1852us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 14:51:54.451488 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 14:51:54.451509 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0927 14:51:54.451749 (+ 240us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 14:51:54.452064 (+ 315us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 14:51:54.452073 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 14:51:54.452075 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 14:51:54.452198 (+ 123us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 14:51:54.452601 (+ 403us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 14:51:54.452608 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 14:51:54.452857 (+ 249us) client_negotiation.cc:769] Sending connection context
0927 14:51:54.453042 (+ 185us) client_negotiation.cc:240] Negotiation successful
0927 14:51:54.453396 (+ 354us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":275,"thread_start_us":113,"threads_started":1}
I20260927 14:51:54.455338 6319 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35495
I20260927 14:51:54.455581 6319 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260927 14:51:54.456092 6319 heartbeater.cc:511] Master 127.0.98.62:35495 requested a full tablet report, sending...
I20260927 14:51:54.457159 5994 ts_manager.cc:194] Registered new tserver with Master: f014825cff914e5aab4ebf3cdf8df37d (127.0.98.2:40739)
I20260927 14:51:54.457765 5994 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:50357
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790520684602251-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790520684602251-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260927 14:51:54.474232 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6936m5/build/debug/bin/kudu
/tmp/dist-test-task6936m5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790520684602251-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790520684602251-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790520684602251-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790520684602251-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35495
--keytab_file=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790520684602251-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36185
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790520684602251-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790520684602251-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790520684602251-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 14:51:54.582170 6324 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 14:51:54.582446 6324 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 14:51:54.582491 6324 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 14:51:54.586633 6324 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 14:51:54.586689 6324 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 14:51:54.586790 6324 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 14:51:54.591995 6324 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36185
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790520684602251-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790520684602251-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790520684602251-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790520684602251-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35495
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6324
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790520684602251-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:15:17 UTC on ccycloud.quasar-ledxfe.root.comops.site
build id 4676134
I20260927 14:51:54.593067 6324 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 14:51:54.594090 6324 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 14:51:54.603921 6330 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 14:51:54.604102 6332 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 14:51:54.604140 6324 server_base.cc:1034] running on GCE node
W20260927 14:51:54.604104 6329 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 14:51:54.604712 6324 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 14:51:54.605367 6324 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 14:51:54.606546 6324 hybrid_clock.cc:648] HybridClock initialized: now 1790520714606533 us; error 27 us; skew 500 ppm
Sep 27 14:51:54 dist-test-slave-xk1g krb5kdc[5949](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790520714, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 14:51:54.609347 6324 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 14:51:54.610635 6324 webserver.cc:460] Webserver started at http://127.0.98.3:40621/ using document root <none> and password file <none>
I20260927 14:51:54.611204 6324 fs_manager.cc:359] Metadata directory not provided
I20260927 14:51:54.611265 6324 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 14:51:54.611481 6324 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 14:51:54.613540 6324 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790520684602251-392-0/minicluster-data/ts-2/data/instance:
uuid: "c90cde66a79b4201bd25c9e799eae659"
format_stamp: "Formatted at 2026-09-27 14:51:54 on dist-test-slave-xk1g"
server_key: "b8c58ff62928fdcdd8a1c89fa04d44b1"
server_key_iv: "6ec34e2370ba122de5d98340bcadd936"
server_key_version: "encryptionkey@0"
I20260927 14:51:54.614084 6324 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790520684602251-392-0/minicluster-data/ts-2/wal/instance:
uuid: "c90cde66a79b4201bd25c9e799eae659"
format_stamp: "Formatted at 2026-09-27 14:51:54 on dist-test-slave-xk1g"
server_key: "b8c58ff62928fdcdd8a1c89fa04d44b1"
server_key_iv: "6ec34e2370ba122de5d98340bcadd936"
server_key_version: "encryptionkey@0"
I20260927 14:51:54.617771 6324 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 14:51:54.620378 6339 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 14:51:54.621706 6324 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 14:51:54.621876 6324 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790520684602251-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790520684602251-392-0/minicluster-data/ts-2/wal
uuid: "c90cde66a79b4201bd25c9e799eae659"
format_stamp: "Formatted at 2026-09-27 14:51:54 on dist-test-slave-xk1g"
server_key: "b8c58ff62928fdcdd8a1c89fa04d44b1"
server_key_iv: "6ec34e2370ba122de5d98340bcadd936"
server_key_version: "encryptionkey@0"
I20260927 14:51:54.621992 6324 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790520684602251-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790520684602251-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790520684602251-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 14:51:54.641809 6324 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 14:51:54.644762 6324 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 14:51:54.644994 6324 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 14:51:54.645637 6324 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 14:51:54.646516 6324 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 14:51:54.646597 6324 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 14:51:54.646677 6324 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 14:51:54.646724 6324 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 14:51:54.656083 6324 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:42095
I20260927 14:51:54.656103 6452 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:42095 every 8 connection(s)
I20260927 14:51:54.657032 6324 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790520684602251-392-0/minicluster-data/ts-2/data/info.pb
I20260927 14:51:54.660629 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6936m5/build/debug/bin/kudu as pid 6324
I20260927 14:51:54.660727 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790520684602251-392-0/minicluster-data/ts-2/wal/instance
I20260927 14:51:54.661008 392 external_mini_cluster.cc:1428] Setting key 92efa5dc0302d7e7f28be2b58a676e9b
Sep 27 14:51:54 dist-test-slave-xk1g krb5kdc[5949](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790520714, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 14:51:54.671021 6033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:51:54.658920 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:42257 (local address 127.0.98.62:35495)
0927 14:51:54.659069 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0927 14:51:54.659073 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 14:51:54.660114 (+ 1041us) server_negotiation.cc:407] Connection header received
0927 14:51:54.661715 (+ 1601us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:51:54.661718 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:51:54.661774 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:51:54.661884 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:51:54.663038 (+ 1154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:51:54.663574 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:51:54.664736 (+ 1162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:51:54.664962 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:51:54.665090 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:51:54.667473 (+ 2383us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:51:54.667477 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:51:54.667482 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 14:51:54.667509 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:51:54.668928 (+ 1419us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:51:54.669487 (+ 559us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:51:54.669492 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:51:54.669494 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:51:54.669582 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:51:54.669929 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:51:54.669932 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:51:54.669934 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:51:54.670118 (+ 184us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:51:54.670225 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0927 14:51:54.670724 (+ 499us) server_negotiation.cc:299] Negotiation successful
0927 14:51:54.670842 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260927 14:51:54.671800 6455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:51:54.659334 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35495 (local address 127.0.98.3:42257)
0927 14:51:54.659960 (+ 626us) negotiation.cc:107] Waiting for socket to connect
0927 14:51:54.660002 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0927 14:51:54.661485 (+ 1483us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:51:54.662064 (+ 579us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:51:54.662071 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:51:54.662421 (+ 350us) client_negotiation.cc:189] Negotiated authn=SASL
0927 14:51:54.662892 (+ 471us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:51:54.662905 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:51:54.663696 (+ 791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:51:54.663701 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:51:54.664577 (+ 876us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:51:54.664586 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:51:54.665129 (+ 543us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:51:54.665133 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:51:54.665299 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:51:54.665916 (+ 617us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 14:51:54.665932 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 14:51:54.667305 (+ 1373us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 14:51:54.669092 (+ 1787us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 14:51:54.669100 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 14:51:54.669115 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0927 14:51:54.669373 (+ 258us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 14:51:54.669676 (+ 303us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 14:51:54.669680 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 14:51:54.669682 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 14:51:54.669825 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 14:51:54.670223 (+ 398us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 14:51:54.670231 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 14:51:54.670485 (+ 254us) client_negotiation.cc:769] Sending connection context
0927 14:51:54.670669 (+ 184us) client_negotiation.cc:240] Negotiation successful
0927 14:51:54.670989 (+ 320us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":444,"thread_start_us":135,"threads_started":1}
I20260927 14:51:54.672907 6453 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35495
I20260927 14:51:54.673174 6453 heartbeater.cc:464] Registering TS with master...
I20260927 14:51:54.673707 6453 heartbeater.cc:511] Master 127.0.98.62:35495 requested a full tablet report, sending...
I20260927 14:51:54.674834 5993 ts_manager.cc:194] Registered new tserver with Master: c90cde66a79b4201bd25c9e799eae659 (127.0.98.3:42095)
I20260927 14:51:54.675433 5993 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:42257
I20260927 14:51:54.686848 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790520684602251-392-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790520684602251-392-0/krb5kdc/test-admin.keytab.
Sep 27 14:51:54 dist-test-slave-xk1g krb5kdc[5949](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790520714, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 14:51:54.702006 392 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260927 14:51:54.704667 6460 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 14:51:54.703336 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57102 (local address 127.0.98.1:45245)
0927 14:51:54.703626 (+ 290us) server_negotiation.cc:206] Beginning negotiation
0927 14:51:54.703641 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 14:51:54.703685 (+ 44us) server_negotiation.cc:407] Connection header received
0927 14:51:54.703735 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:51:54.703739 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:51:54.703926 (+ 187us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:51:54.704039 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:51:54.704476 (+ 437us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57102: BlockingRecv error: recv got EOF from 127.0.0.1:57102 (error 108)
Metrics: {"server-negotiator.queue_time_us":194,"thread_start_us":90,"threads_started":1}
W20260927 14:51:54.704844 392 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790520684602251-392-0/krb5kdc/krb5cc' of type FILE
Sep 27 14:51:54 dist-test-slave-xk1g krb5kdc[5949](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790520714, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 14:51:54.706086 392 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 27 14:51:54 dist-test-slave-xk1g krb5kdc[5949](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790520714, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 14:51:54.707362 392 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 27 14:51:54 dist-test-slave-xk1g krb5kdc[5949](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790520714, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260927 14:51:54.717278 6460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:51:54.708456 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57116 (local address 127.0.98.1:45245)
0927 14:51:54.708586 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0927 14:51:54.708590 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 14:51:54.708605 (+ 15us) server_negotiation.cc:407] Connection header received
0927 14:51:54.708698 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:51:54.708701 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:51:54.708750 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:51:54.708837 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:51:54.709442 (+ 605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:51:54.710328 (+ 886us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:51:54.711249 (+ 921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:51:54.711619 (+ 370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:51:54.711738 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:51:54.714015 (+ 2277us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:51:54.714024 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:51:54.714036 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 14:51:54.714061 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:51:54.715538 (+ 1477us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:51:54.715933 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:51:54.715938 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:51:54.715939 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:51:54.716016 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:51:54.716340 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:51:54.716342 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:51:54.716344 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:51:54.716578 (+ 234us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:51:54.716724 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0927 14:51:54.716944 (+ 220us) server_negotiation.cc:299] Negotiation successful
0927 14:51:54.717127 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
W20260927 14:51:54.720857 6460 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 14:51:54.720012 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57128 (local address 127.0.98.1:45245)
0927 14:51:54.720139 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0927 14:51:54.720142 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 14:51:54.720157 (+ 15us) server_negotiation.cc:407] Connection header received
0927 14:51:54.720259 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:51:54.720261 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:51:54.720296 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:51:54.720358 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:51:54.720755 (+ 397us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57128: BlockingRecv error: recv got EOF from 127.0.0.1:57128 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
W20260927 14:51:54.721014 392 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790520684602251-392-0/krb5kdc/krb5cc' of type FILE
Sep 27 14:51:54 dist-test-slave-xk1g krb5kdc[5949](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790520714, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 14:51:54.721967 392 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 27 14:51:54 dist-test-slave-xk1g krb5kdc[5949](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790520714, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 14:51:54.722832 392 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 27 14:51:54 dist-test-slave-xk1g krb5kdc[5949](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790520714, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260927 14:51:54.731004 6460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:51:54.723549 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57142 (local address 127.0.98.1:45245)
0927 14:51:54.723680 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0927 14:51:54.723683 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 14:51:54.723745 (+ 62us) server_negotiation.cc:407] Connection header received
0927 14:51:54.723828 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:51:54.723830 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:51:54.723860 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:51:54.723917 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:51:54.724317 (+ 400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:51:54.724788 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:51:54.725636 (+ 848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:51:54.725897 (+ 261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:51:54.725992 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:51:54.728348 (+ 2356us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:51:54.728351 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:51:54.728352 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 14:51:54.728374 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:51:54.729650 (+ 1276us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:51:54.730007 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:51:54.730010 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:51:54.730011 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:51:54.730076 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:51:54.730284 (+ 208us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:51:54.730286 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:51:54.730287 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:51:54.730422 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:51:54.730522 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0927 14:51:54.730704 (+ 182us) server_negotiation.cc:299] Negotiation successful
0927 14:51:54.730837 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
W20260927 14:51:54.733506 6460 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 14:51:54.732655 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57146 (local address 127.0.98.1:45245)
0927 14:51:54.732768 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0927 14:51:54.732771 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 14:51:54.732802 (+ 31us) server_negotiation.cc:407] Connection header received
0927 14:51:54.732897 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:51:54.732899 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:51:54.732930 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:51:54.732993 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:51:54.733418 (+ 425us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57146: BlockingRecv error: recv got EOF from 127.0.0.1:57146 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
W20260927 14:51:54.733623 392 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790520684602251-392-0/krb5kdc/krb5cc' of type FILE
Sep 27 14:51:54 dist-test-slave-xk1g krb5kdc[5949](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790520714, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 14:51:54.734622 392 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 27 14:51:54 dist-test-slave-xk1g krb5kdc[5949](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790520714, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 14:51:54.735491 392 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 27 14:51:54 dist-test-slave-xk1g krb5kdc[5949](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790520714, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260927 14:51:54.744266 6460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:51:54.736356 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57160 (local address 127.0.98.1:45245)
0927 14:51:54.736474 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0927 14:51:54.736478 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 14:51:54.736608 (+ 130us) server_negotiation.cc:407] Connection header received
0927 14:51:54.736737 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:51:54.736740 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:51:54.736785 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:51:54.736873 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:51:54.737441 (+ 568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:51:54.737952 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:51:54.738809 (+ 857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:51:54.739029 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:51:54.739139 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:51:54.741586 (+ 2447us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:51:54.741589 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:51:54.741591 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 14:51:54.741612 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:51:54.742907 (+ 1295us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:51:54.743272 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:51:54.743274 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:51:54.743275 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:51:54.743344 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:51:54.743580 (+ 236us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:51:54.743582 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:51:54.743583 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:51:54.743723 (+ 140us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:51:54.743814 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0927 14:51:54.744040 (+ 226us) server_negotiation.cc:299] Negotiation successful
0927 14:51:54.744158 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260927 14:51:54.745208 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6936m5/build/debug/bin/kudu with pid 6056
I20260927 14:51:54.751360 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6936m5/build/debug/bin/kudu with pid 6190
I20260927 14:51:54.756531 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6936m5/build/debug/bin/kudu with pid 6324
I20260927 14:51:54.761605 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6936m5/build/debug/bin/kudu with pid 5962
2026-09-27T14:51:54Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (2174 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790520684602251-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 14:51:54 dist-test-slave-xk1g krb5kdc[6473](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 14:51:54 dist-test-slave-xk1g krb5kdc[6473](info): set up 2 sockets
Sep 27 14:51:54 dist-test-slave-xk1g krb5kdc[6473](info): commencing operation
krb5kdc: starting...
W20260927 14:51:55.853910 392 mini_kdc.cc:121] Time spent starting KDC: real 1.071s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
W20260927 14:51:56.032516 392 mini_kdc.cc:253] Time spent creating user principal test-admin: real 0.178s user 0.001s sys 0.001s
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790520684602251-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790520684602251-392-0/krb5kdc/test-user.keytab.
Sep 27 14:51:56 dist-test-slave-xk1g krb5kdc[6473](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790520716, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T14:51:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T14:51:56Z Disabled control of system clock
WARNING: no policy specified for oryx/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790520684602251-392-0/krb5kdc/oryx_127.0.98.62.keytab.
Entry for principal oryx/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790520684602251-392-0/krb5kdc/oryx_127.0.98.62.keytab.
I20260927 14:51:56.136101 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6936m5/build/debug/bin/kudu
/tmp/dist-test-task6936m5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790520684602251-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790520684602251-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790520684602251-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790520684602251-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36527
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790520684602251-392-0/krb5kdc/oryx_127.0.98.62.keytab
--principal=oryx/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44229
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:36527
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790520684602251-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790520684602251-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790520684602251-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 14:51:56.244028 6486 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 14:51:56.244310 6486 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 14:51:56.244356 6486 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 14:51:56.248253 6486 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 14:51:56.248313 6486 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 14:51:56.248338 6486 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 14:51:56.248358 6486 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 14:51:56.248377 6486 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 14:51:56.253564 6486 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44229
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790520684602251-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790520684602251-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:36527
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790520684602251-392-0/krb5kdc/oryx_127.0.98.62.keytab
--principal=oryx/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36527
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790520684602251-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6486
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790520684602251-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:15:17 UTC on ccycloud.quasar-ledxfe.root.comops.site
build id 4676134
I20260927 14:51:56.254527 6486 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 14:51:56.255473 6486 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 14:51:56.263860 6492 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 14:51:56.263963 6486 server_base.cc:1034] running on GCE node
W20260927 14:51:56.263911 6491 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 14:51:56.263912 6494 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 14:51:56.264719 6486 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 14:51:56.265868 6486 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 14:51:56.267026 6486 hybrid_clock.cc:648] HybridClock initialized: now 1790520716267008 us; error 31 us; skew 500 ppm
Sep 27 14:51:56 dist-test-slave-xk1g krb5kdc[6473](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790520716, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 14:51:56.269651 6486 init.cc:377] Logged in from keytab as oryx/127.0.98.62@KRBTEST.COM (short username oryx)
I20260927 14:51:56.270503 6486 webserver.cc:460] Webserver started at http://127.0.98.62:35639/ using document root <none> and password file <none>
I20260927 14:51:56.271054 6486 fs_manager.cc:359] Metadata directory not provided
I20260927 14:51:56.271143 6486 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 14:51:56.271369 6486 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 14:51:56.273413 6486 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790520684602251-392-0/minicluster-data/master-0/data/instance:
uuid: "d60b79f337e343bca73eadfa1bd55f6a"
format_stamp: "Formatted at 2026-09-27 14:51:56 on dist-test-slave-xk1g"
server_key: "f8407c82427d26c347225da10fc0d451"
server_key_iv: "bc4aaf7fa314bce0d1661014153dc1dc"
server_key_version: "encryptionkey@0"
I20260927 14:51:56.273969 6486 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790520684602251-392-0/minicluster-data/master-0/wal/instance:
uuid: "d60b79f337e343bca73eadfa1bd55f6a"
format_stamp: "Formatted at 2026-09-27 14:51:56 on dist-test-slave-xk1g"
server_key: "f8407c82427d26c347225da10fc0d451"
server_key_iv: "bc4aaf7fa314bce0d1661014153dc1dc"
server_key_version: "encryptionkey@0"
I20260927 14:51:56.277493 6486 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260927 14:51:56.279838 6501 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 14:51:56.281152 6486 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 14:51:56.281281 6486 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790520684602251-392-0/minicluster-data/master-0/data,/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790520684602251-392-0/minicluster-data/master-0/wal
uuid: "d60b79f337e343bca73eadfa1bd55f6a"
format_stamp: "Formatted at 2026-09-27 14:51:56 on dist-test-slave-xk1g"
server_key: "f8407c82427d26c347225da10fc0d451"
server_key_iv: "bc4aaf7fa314bce0d1661014153dc1dc"
server_key_version: "encryptionkey@0"
I20260927 14:51:56.281390 6486 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790520684602251-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790520684602251-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790520684602251-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 14:51:56.303046 6486 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 14:51:56.305788 6486 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 14:51:56.305990 6486 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 14:51:56.312674 6486 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:36527
I20260927 14:51:56.312965 6553 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:36527 every 8 connection(s)
I20260927 14:51:56.313652 6486 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790520684602251-392-0/minicluster-data/master-0/data/info.pb
I20260927 14:51:56.316278 6554 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 14:51:56.321712 6554 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d60b79f337e343bca73eadfa1bd55f6a: Bootstrap starting.
I20260927 14:51:56.321898 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6936m5/build/debug/bin/kudu as pid 6486
I20260927 14:51:56.322031 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790520684602251-392-0/minicluster-data/master-0/wal/instance
I20260927 14:51:56.322285 392 external_mini_cluster.cc:1428] Setting key d26a56a868570ce96d08778b25eafe7b
I20260927 14:51:56.324455 6554 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d60b79f337e343bca73eadfa1bd55f6a: Neither blocks nor log segments found. Creating new log.
I20260927 14:51:56.325253 6554 log.cc:824] T 00000000000000000000000000000000 P d60b79f337e343bca73eadfa1bd55f6a: Log is configured to *not* fsync() on all Append() calls
I20260927 14:51:56.327100 6554 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d60b79f337e343bca73eadfa1bd55f6a: No bootstrap required, opened a new log
I20260927 14:51:56.330852 6554 raft_consensus.cc:348] T 00000000000000000000000000000000 P d60b79f337e343bca73eadfa1bd55f6a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d60b79f337e343bca73eadfa1bd55f6a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36527 } }
I20260927 14:51:56.331144 6554 raft_consensus.cc:374] T 00000000000000000000000000000000 P d60b79f337e343bca73eadfa1bd55f6a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 14:51:56.331195 6554 raft_consensus.cc:729] T 00000000000000000000000000000000 P d60b79f337e343bca73eadfa1bd55f6a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d60b79f337e343bca73eadfa1bd55f6a, State: Initialized, Role: FOLLOWER
Sep 27 14:51:56 dist-test-slave-xk1g krb5kdc[6473](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790520716, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260927 14:51:56.331677 6554 consensus_queue.cc:260] T 00000000000000000000000000000000 P d60b79f337e343bca73eadfa1bd55f6a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d60b79f337e343bca73eadfa1bd55f6a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36527 } }
I20260927 14:51:56.331835 6554 raft_consensus.cc:388] T 00000000000000000000000000000000 P d60b79f337e343bca73eadfa1bd55f6a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 14:51:56.331903 6554 raft_consensus.cc:482] T 00000000000000000000000000000000 P d60b79f337e343bca73eadfa1bd55f6a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 14:51:56.332015 6554 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d60b79f337e343bca73eadfa1bd55f6a [term 0 FOLLOWER]: Advancing to term 1
I20260927 14:51:56.333166 6554 raft_consensus.cc:504] T 00000000000000000000000000000000 P d60b79f337e343bca73eadfa1bd55f6a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d60b79f337e343bca73eadfa1bd55f6a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36527 } }
I20260927 14:51:56.333514 6554 leader_election.cc:302] T 00000000000000000000000000000000 P d60b79f337e343bca73eadfa1bd55f6a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d60b79f337e343bca73eadfa1bd55f6a; no voters:
I20260927 14:51:56.333849 6554 leader_election.cc:288] T 00000000000000000000000000000000 P d60b79f337e343bca73eadfa1bd55f6a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 14:51:56.333945 6559 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d60b79f337e343bca73eadfa1bd55f6a [term 1 FOLLOWER]: Leader election won for term 1
I20260927 14:51:56.334167 6559 raft_consensus.cc:686] T 00000000000000000000000000000000 P d60b79f337e343bca73eadfa1bd55f6a [term 1 LEADER]: Becoming Leader. State: Replica: d60b79f337e343bca73eadfa1bd55f6a, State: Running, Role: LEADER
I20260927 14:51:56.334570 6559 consensus_queue.cc:237] T 00000000000000000000000000000000 P d60b79f337e343bca73eadfa1bd55f6a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d60b79f337e343bca73eadfa1bd55f6a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36527 } }
I20260927 14:51:56.335146 6554 sys_catalog.cc:564] T 00000000000000000000000000000000 P d60b79f337e343bca73eadfa1bd55f6a [sys.catalog]: configured and running, proceeding with master startup.
I20260927 14:51:56.335800 6561 sys_catalog.cc:455] T 00000000000000000000000000000000 P d60b79f337e343bca73eadfa1bd55f6a [sys.catalog]: SysCatalogTable state changed. Reason: New leader d60b79f337e343bca73eadfa1bd55f6a. Latest consensus state: current_term: 1 leader_uuid: "d60b79f337e343bca73eadfa1bd55f6a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d60b79f337e343bca73eadfa1bd55f6a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36527 } } }
I20260927 14:51:56.336560 6561 sys_catalog.cc:458] T 00000000000000000000000000000000 P d60b79f337e343bca73eadfa1bd55f6a [sys.catalog]: This master's current role is: LEADER
I20260927 14:51:56.336457 6560 sys_catalog.cc:455] T 00000000000000000000000000000000 P d60b79f337e343bca73eadfa1bd55f6a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d60b79f337e343bca73eadfa1bd55f6a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d60b79f337e343bca73eadfa1bd55f6a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36527 } } }
I20260927 14:51:56.336968 6560 sys_catalog.cc:458] T 00000000000000000000000000000000 P d60b79f337e343bca73eadfa1bd55f6a [sys.catalog]: This master's current role is: LEADER
I20260927 14:51:56.337754 6557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:51:56.323602 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55352 (local address 127.0.98.62:36527)
0927 14:51:56.324069 (+ 467us) server_negotiation.cc:206] Beginning negotiation
0927 14:51:56.324084 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 14:51:56.324123 (+ 39us) server_negotiation.cc:407] Connection header received
0927 14:51:56.324859 (+ 736us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:51:56.324878 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:51:56.325300 (+ 422us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:51:56.325677 (+ 377us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:51:56.326483 (+ 806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:51:56.327586 (+ 1103us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:51:56.328429 (+ 843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:51:56.328907 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:51:56.329110 (+ 203us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:51:56.331878 (+ 2768us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:51:56.331887 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:51:56.331909 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 14:51:56.331954 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:51:56.334040 (+ 2086us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:51:56.334565 (+ 525us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:51:56.334570 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:51:56.334577 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:51:56.334714 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:51:56.335018 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:51:56.335021 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:51:56.335023 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:51:56.335498 (+ 475us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:51:56.335708 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0927 14:51:56.336334 (+ 626us) server_negotiation.cc:299] Negotiation successful
0927 14:51:56.336667 (+ 333us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":4,"server-negotiator.queue_time_us":325,"thread_start_us":142,"threads_started":1}
I20260927 14:51:56.337808 6571 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 14:51:56.341046 6571 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 14:51:56.346261 6571 catalog_manager.cc:1347] Generated new cluster ID: 3ea1e4dfc1684ee8ba96f40d83c0eb15
I20260927 14:51:56.346408 6571 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 14:51:56.351958 6571 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 14:51:56.352843 6571 catalog_manager.cc:1504] Loading token signing keys...
I20260927 14:51:56.360077 6571 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d60b79f337e343bca73eadfa1bd55f6a: Generated new TSK 0
I20260927 14:51:56.360785 6571 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790520684602251-392-0/krb5kdc/oryx_127.0.98.1.keytab.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790520684602251-392-0/krb5kdc/oryx_127.0.98.1.keytab.
I20260927 14:51:56.391919 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6936m5/build/debug/bin/kudu
/tmp/dist-test-task6936m5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790520684602251-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790520684602251-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790520684602251-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790520684602251-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36527
--keytab_file=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790520684602251-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44229
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790520684602251-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790520684602251-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790520684602251-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 14:51:56.505354 6580 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 14:51:56.505617 6580 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 14:51:56.505661 6580 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 14:51:56.509728 6580 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 14:51:56.509781 6580 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 14:51:56.509837 6580 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 14:51:56.515003 6580 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44229
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790520684602251-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790520684602251-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790520684602251-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790520684602251-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36527
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6580
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790520684602251-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:15:17 UTC on ccycloud.quasar-ledxfe.root.comops.site
build id 4676134
I20260927 14:51:56.515996 6580 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 14:51:56.516973 6580 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 14:51:56.526571 6588 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 14:51:56.526559 6586 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 14:51:56.526540 6585 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 14:51:56.526741 6580 server_base.cc:1034] running on GCE node
I20260927 14:51:56.527136 6580 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 14:51:56.527827 6580 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 14:51:56.529031 6580 hybrid_clock.cc:648] HybridClock initialized: now 1790520716529019 us; error 29 us; skew 500 ppm
Sep 27 14:51:56 dist-test-slave-xk1g krb5kdc[6473](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790520716, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 14:51:56.531781 6580 init.cc:377] Logged in from keytab as oryx/127.0.98.1@KRBTEST.COM (short username oryx)
I20260927 14:51:56.532727 6580 webserver.cc:460] Webserver started at http://127.0.98.1:33431/ using document root <none> and password file <none>
I20260927 14:51:56.533368 6580 fs_manager.cc:359] Metadata directory not provided
I20260927 14:51:56.533465 6580 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 14:51:56.533708 6580 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 14:51:56.535808 6580 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790520684602251-392-0/minicluster-data/ts-0/data/instance:
uuid: "f37a65090f8b4b8f9a4facdb930c9eec"
format_stamp: "Formatted at 2026-09-27 14:51:56 on dist-test-slave-xk1g"
server_key: "814b5ad532276f62ea9f3dc83580cd70"
server_key_iv: "d3a228153c70238fb4e9d69303bde9c5"
server_key_version: "encryptionkey@0"
I20260927 14:51:56.536444 6580 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790520684602251-392-0/minicluster-data/ts-0/wal/instance:
uuid: "f37a65090f8b4b8f9a4facdb930c9eec"
format_stamp: "Formatted at 2026-09-27 14:51:56 on dist-test-slave-xk1g"
server_key: "814b5ad532276f62ea9f3dc83580cd70"
server_key_iv: "d3a228153c70238fb4e9d69303bde9c5"
server_key_version: "encryptionkey@0"
I20260927 14:51:56.540263 6580 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.001s
I20260927 14:51:56.542752 6595 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 14:51:56.543969 6580 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 14:51:56.544099 6580 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790520684602251-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790520684602251-392-0/minicluster-data/ts-0/wal
uuid: "f37a65090f8b4b8f9a4facdb930c9eec"
format_stamp: "Formatted at 2026-09-27 14:51:56 on dist-test-slave-xk1g"
server_key: "814b5ad532276f62ea9f3dc83580cd70"
server_key_iv: "d3a228153c70238fb4e9d69303bde9c5"
server_key_version: "encryptionkey@0"
I20260927 14:51:56.544206 6580 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790520684602251-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790520684602251-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790520684602251-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 14:51:56.555084 6580 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 14:51:56.557936 6580 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 14:51:56.558143 6580 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 14:51:56.558743 6580 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 14:51:56.559574 6580 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 14:51:56.559621 6580 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 14:51:56.559693 6580 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 14:51:56.559724 6580 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 14:51:56.568841 6580 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:43017
I20260927 14:51:56.568868 6708 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:43017 every 8 connection(s)
I20260927 14:51:56.569794 6580 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790520684602251-392-0/minicluster-data/ts-0/data/info.pb
I20260927 14:51:56.578042 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6936m5/build/debug/bin/kudu as pid 6580
I20260927 14:51:56.578172 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790520684602251-392-0/minicluster-data/ts-0/wal/instance
I20260927 14:51:56.578469 392 external_mini_cluster.cc:1428] Setting key ab6170ff180d4548c0b517e21faae75a
Sep 27 14:51:56 dist-test-slave-xk1g krb5kdc[6473](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790520716, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260927 14:51:56.583686 6557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:51:56.571539 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:56905 (local address 127.0.98.62:36527)
0927 14:51:56.571671 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0927 14:51:56.571674 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 14:51:56.572524 (+ 850us) server_negotiation.cc:407] Connection header received
0927 14:51:56.573858 (+ 1334us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:51:56.573861 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:51:56.573906 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:51:56.573984 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:51:56.575141 (+ 1157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:51:56.575671 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:51:56.576817 (+ 1146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:51:56.577112 (+ 295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:51:56.577224 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:51:56.580099 (+ 2875us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:51:56.580104 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:51:56.580107 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 14:51:56.580135 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:51:56.581652 (+ 1517us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:51:56.582178 (+ 526us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:51:56.582182 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:51:56.582184 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:51:56.582268 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:51:56.582604 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:51:56.582609 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:51:56.582611 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:51:56.582791 (+ 180us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:51:56.582895 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0927 14:51:56.583378 (+ 483us) server_negotiation.cc:299] Negotiation successful
0927 14:51:56.583522 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260927 14:51:56.584450 6711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:51:56.571957 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36527 (local address 127.0.98.1:56905)
0927 14:51:56.572373 (+ 416us) negotiation.cc:107] Waiting for socket to connect
0927 14:51:56.572414 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0927 14:51:56.573662 (+ 1248us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:51:56.574143 (+ 481us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:51:56.574150 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:51:56.574501 (+ 351us) client_negotiation.cc:189] Negotiated authn=SASL
0927 14:51:56.574968 (+ 467us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:51:56.574982 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:51:56.575798 (+ 816us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:51:56.575802 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:51:56.576686 (+ 884us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:51:56.576695 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:51:56.577258 (+ 563us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:51:56.577262 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:51:56.577430 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:51:56.578033 (+ 603us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 14:51:56.578049 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 14:51:56.579917 (+ 1868us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 14:51:56.581774 (+ 1857us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 14:51:56.581779 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 14:51:56.581793 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0927 14:51:56.582072 (+ 279us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 14:51:56.582367 (+ 295us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 14:51:56.582370 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 14:51:56.582372 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 14:51:56.582510 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 14:51:56.582905 (+ 395us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 14:51:56.582913 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 14:51:56.583154 (+ 241us) client_negotiation.cc:769] Sending connection context
0927 14:51:56.583342 (+ 188us) client_negotiation.cc:240] Negotiation successful
0927 14:51:56.583667 (+ 325us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":242,"thread_start_us":105,"threads_started":1}
I20260927 14:51:56.585736 6709 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36527
I20260927 14:51:56.585979 6709 heartbeater.cc:464] Registering TS with master...
I20260927 14:51:56.586422 6709 heartbeater.cc:511] Master 127.0.98.62:36527 requested a full tablet report, sending...
WARNING: no policy specified for oryx/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260927 14:51:56.588084 6518 ts_manager.cc:194] Registered new tserver with Master: f37a65090f8b4b8f9a4facdb930c9eec (127.0.98.1:43017)
I20260927 14:51:56.589327 6518 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:56905
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790520684602251-392-0/krb5kdc/oryx_127.0.98.2.keytab.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790520684602251-392-0/krb5kdc/oryx_127.0.98.2.keytab.
I20260927 14:51:56.605636 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6936m5/build/debug/bin/kudu
/tmp/dist-test-task6936m5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790520684602251-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790520684602251-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790520684602251-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790520684602251-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36527
--keytab_file=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790520684602251-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44229
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790520684602251-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790520684602251-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790520684602251-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 14:51:56.711551 6714 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 14:51:56.711789 6714 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 14:51:56.711835 6714 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 14:51:56.716945 6714 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 14:51:56.716996 6714 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 14:51:56.717049 6714 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 14:51:56.722040 6714 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44229
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790520684602251-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790520684602251-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790520684602251-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790520684602251-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36527
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6714
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790520684602251-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:15:17 UTC on ccycloud.quasar-ledxfe.root.comops.site
build id 4676134
I20260927 14:51:56.722904 6714 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 14:51:56.723783 6714 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 14:51:56.732894 6722 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 14:51:56.732894 6720 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 14:51:56.733193 6714 server_base.cc:1034] running on GCE node
W20260927 14:51:56.733106 6719 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 14:51:56.733564 6714 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 14:51:56.734184 6714 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 14:51:56.735371 6714 hybrid_clock.cc:648] HybridClock initialized: now 1790520716735355 us; error 35 us; skew 500 ppm
Sep 27 14:51:56 dist-test-slave-xk1g krb5kdc[6473](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790520716, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 14:51:56.738116 6714 init.cc:377] Logged in from keytab as oryx/127.0.98.2@KRBTEST.COM (short username oryx)
I20260927 14:51:56.738837 6714 webserver.cc:460] Webserver started at http://127.0.98.2:36055/ using document root <none> and password file <none>
I20260927 14:51:56.739318 6714 fs_manager.cc:359] Metadata directory not provided
I20260927 14:51:56.739373 6714 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 14:51:56.739526 6714 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 14:51:56.741595 6714 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790520684602251-392-0/minicluster-data/ts-1/data/instance:
uuid: "a473db7856d24a8ba3efb9bbc9e12de9"
format_stamp: "Formatted at 2026-09-27 14:51:56 on dist-test-slave-xk1g"
server_key: "2d2692160a00ee3fd176e4475d2e6a41"
server_key_iv: "db5eaf6ce4e6f481305152d0e6b40338"
server_key_version: "encryptionkey@0"
I20260927 14:51:56.742090 6714 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790520684602251-392-0/minicluster-data/ts-1/wal/instance:
uuid: "a473db7856d24a8ba3efb9bbc9e12de9"
format_stamp: "Formatted at 2026-09-27 14:51:56 on dist-test-slave-xk1g"
server_key: "2d2692160a00ee3fd176e4475d2e6a41"
server_key_iv: "db5eaf6ce4e6f481305152d0e6b40338"
server_key_version: "encryptionkey@0"
I20260927 14:51:56.745491 6714 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 14:51:56.747713 6729 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 14:51:56.748989 6714 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 14:51:56.749127 6714 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790520684602251-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790520684602251-392-0/minicluster-data/ts-1/wal
uuid: "a473db7856d24a8ba3efb9bbc9e12de9"
format_stamp: "Formatted at 2026-09-27 14:51:56 on dist-test-slave-xk1g"
server_key: "2d2692160a00ee3fd176e4475d2e6a41"
server_key_iv: "db5eaf6ce4e6f481305152d0e6b40338"
server_key_version: "encryptionkey@0"
I20260927 14:51:56.749214 6714 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790520684602251-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790520684602251-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790520684602251-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 14:51:56.759927 6714 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 14:51:56.763134 6714 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 14:51:56.763279 6714 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 14:51:56.763792 6714 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 14:51:56.764520 6714 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 14:51:56.764572 6714 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 14:51:56.764609 6714 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 14:51:56.764624 6714 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 14:51:56.773250 6714 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:34621
I20260927 14:51:56.773324 6842 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:34621 every 8 connection(s)
I20260927 14:51:56.774163 6714 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790520684602251-392-0/minicluster-data/ts-1/data/info.pb
I20260927 14:51:56.781302 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6936m5/build/debug/bin/kudu as pid 6714
I20260927 14:51:56.781440 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790520684602251-392-0/minicluster-data/ts-1/wal/instance
I20260927 14:51:56.781761 392 external_mini_cluster.cc:1428] Setting key 070cb83c202ac415fb5cce6d7704406b
Sep 27 14:51:56 dist-test-slave-xk1g krb5kdc[6473](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790520716, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260927 14:51:56.787891 6557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:51:56.775945 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:35353 (local address 127.0.98.62:36527)
0927 14:51:56.776097 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0927 14:51:56.776101 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 14:51:56.776873 (+ 772us) server_negotiation.cc:407] Connection header received
0927 14:51:56.778206 (+ 1333us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:51:56.778209 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:51:56.778255 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:51:56.778331 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:51:56.779459 (+ 1128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:51:56.779972 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:51:56.781109 (+ 1137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:51:56.781350 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:51:56.781493 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:51:56.784183 (+ 2690us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:51:56.784186 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:51:56.784188 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 14:51:56.784212 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:51:56.785941 (+ 1729us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:51:56.786403 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:51:56.786406 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:51:56.786409 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:51:56.786516 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:51:56.786849 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:51:56.786853 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:51:56.786855 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:51:56.787047 (+ 192us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:51:56.787155 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0927 14:51:56.787625 (+ 470us) server_negotiation.cc:299] Negotiation successful
0927 14:51:56.787738 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260927 14:51:56.788692 6845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:51:56.776312 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36527 (local address 127.0.98.2:35353)
0927 14:51:56.776722 (+ 410us) negotiation.cc:107] Waiting for socket to connect
0927 14:51:56.776761 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0927 14:51:56.778018 (+ 1257us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:51:56.778488 (+ 470us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:51:56.778495 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:51:56.778823 (+ 328us) client_negotiation.cc:189] Negotiated authn=SASL
0927 14:51:56.779287 (+ 464us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:51:56.779300 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:51:56.780075 (+ 775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:51:56.780078 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:51:56.780917 (+ 839us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:51:56.780926 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:51:56.781561 (+ 635us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:51:56.781566 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:51:56.781765 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:51:56.782434 (+ 669us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 14:51:56.782465 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0927 14:51:56.783986 (+ 1521us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 14:51:56.786050 (+ 2064us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 14:51:56.786055 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 14:51:56.786066 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0927 14:51:56.786307 (+ 241us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 14:51:56.786617 (+ 310us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 14:51:56.786619 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 14:51:56.786620 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 14:51:56.786743 (+ 123us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 14:51:56.787152 (+ 409us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 14:51:56.787159 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 14:51:56.787390 (+ 231us) client_negotiation.cc:769] Sending connection context
0927 14:51:56.787586 (+ 196us) client_negotiation.cc:240] Negotiation successful
0927 14:51:56.787895 (+ 309us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":238,"thread_start_us":109,"threads_started":1}
I20260927 14:51:56.789994 6843 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36527
I20260927 14:51:56.790230 6843 heartbeater.cc:464] Registering TS with master...
I20260927 14:51:56.790685 6843 heartbeater.cc:511] Master 127.0.98.62:36527 requested a full tablet report, sending...
I20260927 14:51:56.791706 6518 ts_manager.cc:194] Registered new tserver with Master: a473db7856d24a8ba3efb9bbc9e12de9 (127.0.98.2:34621)
WARNING: no policy specified for oryx/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260927 14:51:56.792296 6518 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:35353
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790520684602251-392-0/krb5kdc/oryx_127.0.98.3.keytab.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790520684602251-392-0/krb5kdc/oryx_127.0.98.3.keytab.
I20260927 14:51:56.808465 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6936m5/build/debug/bin/kudu
/tmp/dist-test-task6936m5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790520684602251-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790520684602251-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790520684602251-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790520684602251-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36527
--keytab_file=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790520684602251-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44229
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790520684602251-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790520684602251-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790520684602251-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 14:51:56.916836 6848 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 14:51:56.917096 6848 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 14:51:56.917137 6848 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 14:51:56.921001 6848 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 14:51:56.921097 6848 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 14:51:56.921206 6848 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 14:51:56.926328 6848 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44229
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790520684602251-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790520684602251-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790520684602251-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790520684602251-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36527
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6848
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790520684602251-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:15:17 UTC on ccycloud.quasar-ledxfe.root.comops.site
build id 4676134
I20260927 14:51:56.927361 6848 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 14:51:56.928341 6848 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 14:51:56.937806 6853 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 14:51:56.937817 6856 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 14:51:56.937829 6854 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 14:51:56.938191 6848 server_base.cc:1034] running on GCE node
I20260927 14:51:56.938566 6848 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 14:51:56.939210 6848 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 14:51:56.940393 6848 hybrid_clock.cc:648] HybridClock initialized: now 1790520716940386 us; error 31 us; skew 500 ppm
Sep 27 14:51:56 dist-test-slave-xk1g krb5kdc[6473](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790520716, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 14:51:56.943262 6848 init.cc:377] Logged in from keytab as oryx/127.0.98.3@KRBTEST.COM (short username oryx)
I20260927 14:51:56.944104 6848 webserver.cc:460] Webserver started at http://127.0.98.3:37503/ using document root <none> and password file <none>
I20260927 14:51:56.944649 6848 fs_manager.cc:359] Metadata directory not provided
I20260927 14:51:56.944710 6848 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 14:51:56.944958 6848 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 14:51:56.946938 6848 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790520684602251-392-0/minicluster-data/ts-2/data/instance:
uuid: "df8799b0da0c4b4e895000a21761211e"
format_stamp: "Formatted at 2026-09-27 14:51:56 on dist-test-slave-xk1g"
server_key: "9aa5d34d8927738d9765447845d19769"
server_key_iv: "32cc3ec66166373ee283f488018464c0"
server_key_version: "encryptionkey@0"
I20260927 14:51:56.947575 6848 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790520684602251-392-0/minicluster-data/ts-2/wal/instance:
uuid: "df8799b0da0c4b4e895000a21761211e"
format_stamp: "Formatted at 2026-09-27 14:51:56 on dist-test-slave-xk1g"
server_key: "9aa5d34d8927738d9765447845d19769"
server_key_iv: "32cc3ec66166373ee283f488018464c0"
server_key_version: "encryptionkey@0"
I20260927 14:51:56.951254 6848 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260927 14:51:56.953675 6863 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 14:51:56.954906 6848 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 14:51:56.955024 6848 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790520684602251-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790520684602251-392-0/minicluster-data/ts-2/wal
uuid: "df8799b0da0c4b4e895000a21761211e"
format_stamp: "Formatted at 2026-09-27 14:51:56 on dist-test-slave-xk1g"
server_key: "9aa5d34d8927738d9765447845d19769"
server_key_iv: "32cc3ec66166373ee283f488018464c0"
server_key_version: "encryptionkey@0"
I20260927 14:51:56.955124 6848 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790520684602251-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790520684602251-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790520684602251-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 14:51:56.968137 6848 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 14:51:56.971235 6848 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 14:51:56.971431 6848 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 14:51:56.971992 6848 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 14:51:56.972761 6848 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 14:51:56.972810 6848 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 14:51:56.972847 6848 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 14:51:56.972900 6848 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 14:51:56.981392 6848 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:38555
I20260927 14:51:56.981472 6976 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:38555 every 8 connection(s)
I20260927 14:51:56.982290 6848 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790520684602251-392-0/minicluster-data/ts-2/data/info.pb
I20260927 14:51:56.984144 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6936m5/build/debug/bin/kudu as pid 6848
I20260927 14:51:56.984279 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790520684602251-392-0/minicluster-data/ts-2/wal/instance
I20260927 14:51:56.984553 392 external_mini_cluster.cc:1428] Setting key b08ff967a30d59a7bd4f6e526ffbbd43
Sep 27 14:51:56 dist-test-slave-xk1g krb5kdc[6473](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790520716, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260927 14:51:56.996060 6557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:51:56.984099 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:43515 (local address 127.0.98.62:36527)
0927 14:51:56.984243 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0927 14:51:56.984247 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 14:51:56.985274 (+ 1027us) server_negotiation.cc:407] Connection header received
0927 14:51:56.986983 (+ 1709us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:51:56.986986 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:51:56.987030 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:51:56.987097 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:51:56.988193 (+ 1096us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:51:56.988702 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:51:56.989922 (+ 1220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:51:56.990139 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:51:56.990216 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:51:56.992522 (+ 2306us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:51:56.992525 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:51:56.992527 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 14:51:56.992549 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:51:56.994014 (+ 1465us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:51:56.994539 (+ 525us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:51:56.994543 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:51:56.994545 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:51:56.994623 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:51:56.994973 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:51:56.994977 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:51:56.994978 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:51:56.995155 (+ 177us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:51:56.995259 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0927 14:51:56.995755 (+ 496us) server_negotiation.cc:299] Negotiation successful
0927 14:51:56.995875 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260927 14:51:56.996834 6979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:51:56.984562 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36527 (local address 127.0.98.3:43515)
0927 14:51:56.985102 (+ 540us) negotiation.cc:107] Waiting for socket to connect
0927 14:51:56.985157 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0927 14:51:56.986734 (+ 1577us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:51:56.987250 (+ 516us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:51:56.987257 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:51:56.987609 (+ 352us) client_negotiation.cc:189] Negotiated authn=SASL
0927 14:51:56.988071 (+ 462us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:51:56.988083 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:51:56.988832 (+ 749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:51:56.988836 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:51:56.989808 (+ 972us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:51:56.989818 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:51:56.990264 (+ 446us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:51:56.990267 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:51:56.990420 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:51:56.990930 (+ 510us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 14:51:56.990945 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 14:51:56.992342 (+ 1397us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 14:51:56.994142 (+ 1800us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 14:51:56.994148 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 14:51:56.994163 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0927 14:51:56.994418 (+ 255us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 14:51:56.994718 (+ 300us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 14:51:56.994721 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 14:51:56.994723 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 14:51:56.994869 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 14:51:56.995257 (+ 388us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 14:51:56.995264 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 14:51:56.995515 (+ 251us) client_negotiation.cc:769] Sending connection context
0927 14:51:56.995702 (+ 187us) client_negotiation.cc:240] Negotiation successful
0927 14:51:56.996026 (+ 324us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":302,"thread_start_us":137,"threads_started":1}
I20260927 14:51:56.998099 6977 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36527
I20260927 14:51:56.998355 6977 heartbeater.cc:464] Registering TS with master...
I20260927 14:51:56.998916 6977 heartbeater.cc:511] Master 127.0.98.62:36527 requested a full tablet report, sending...
I20260927 14:51:56.999924 6518 ts_manager.cc:194] Registered new tserver with Master: df8799b0da0c4b4e895000a21761211e (127.0.98.3:38555)
I20260927 14:51:57.000455 6518 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:43515
I20260927 14:51:57.010754 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.0.98.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790520684602251-392-0/krb5kdc/oryx_127.0.98.61.keytab.
Entry for principal oryx/127.0.98.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790520684602251-392-0/krb5kdc/oryx_127.0.98.61.keytab.
I20260927 14:51:57.037209 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6936m5/build/debug/bin/kudu
/tmp/dist-test-task6936m5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790520684602251-392-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790520684602251-392-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790520684602251-392-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790520684602251-392-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:37119
--webserver_interface=127.0.98.61
--webserver_port=0
--keytab_file=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790520684602251-392-0/krb5kdc/oryx_127.0.98.61.keytab
--principal=oryx/127.0.98.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44229
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:36527,127.0.98.61:37119
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790520684602251-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790520684602251-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790520684602251-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 14:51:57.146656 6982 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 14:51:57.146947 6982 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 14:51:57.147037 6982 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 14:51:57.151064 6982 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 14:51:57.151114 6982 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 14:51:57.151134 6982 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 14:51:57.151149 6982 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 14:51:57.151197 6982 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 14:51:57.156291 6982 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44229
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790520684602251-392-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790520684602251-392-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:36527,127.0.98.61:37119
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790520684602251-392-0/krb5kdc/oryx_127.0.98.61.keytab
--principal=oryx/127.0.98.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:37119
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790520684602251-392-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.98.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6982
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790520684602251-392-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:15:17 UTC on ccycloud.quasar-ledxfe.root.comops.site
build id 4676134
W20260927 14:51:57.156492 6982 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.0.98.62:36527,127.0.98.61:37119'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260927 14:51:57.157299 6982 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 14:51:57.158246 6982 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 14:51:57.166275 6988 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 14:51:57.166280 6990 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 14:51:57.166368 6987 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 14:51:57.166503 6982 server_base.cc:1034] running on GCE node
I20260927 14:51:57.167122 6982 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 14:51:57.168103 6982 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 14:51:57.169253 6982 hybrid_clock.cc:648] HybridClock initialized: now 1790520717169244 us; error 29 us; skew 500 ppm
Sep 27 14:51:57 dist-test-slave-xk1g krb5kdc[6473](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790520717, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 14:51:57.171878 6982 init.cc:377] Logged in from keytab as oryx/127.0.98.61@KRBTEST.COM (short username oryx)
I20260927 14:51:57.172780 6982 webserver.cc:460] Webserver started at http://127.0.98.61:39987/ using document root <none> and password file <none>
I20260927 14:51:57.173338 6982 fs_manager.cc:359] Metadata directory not provided
I20260927 14:51:57.173399 6982 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 14:51:57.173566 6982 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 14:51:57.175462 6982 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790520684602251-392-0/minicluster-data/master-1/data/instance:
uuid: "413c118097b64cd8af3711ec05bf636a"
format_stamp: "Formatted at 2026-09-27 14:51:57 on dist-test-slave-xk1g"
server_key: "0ff963f6b0c0f7368489baa55ad85b67"
server_key_iv: "8538639a3832603e4d21c174afa7eeeb"
server_key_version: "encryptionkey@0"
I20260927 14:51:57.175961 6982 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790520684602251-392-0/minicluster-data/master-1/wal/instance:
uuid: "413c118097b64cd8af3711ec05bf636a"
format_stamp: "Formatted at 2026-09-27 14:51:57 on dist-test-slave-xk1g"
server_key: "0ff963f6b0c0f7368489baa55ad85b67"
server_key_iv: "8538639a3832603e4d21c174afa7eeeb"
server_key_version: "encryptionkey@0"
I20260927 14:51:57.179520 6982 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260927 14:51:57.181854 6997 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 14:51:57.183131 6982 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 14:51:57.183229 6982 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790520684602251-392-0/minicluster-data/master-1/data,/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790520684602251-392-0/minicluster-data/master-1/wal
uuid: "413c118097b64cd8af3711ec05bf636a"
format_stamp: "Formatted at 2026-09-27 14:51:57 on dist-test-slave-xk1g"
server_key: "0ff963f6b0c0f7368489baa55ad85b67"
server_key_iv: "8538639a3832603e4d21c174afa7eeeb"
server_key_version: "encryptionkey@0"
I20260927 14:51:57.183313 6982 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790520684602251-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790520684602251-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790520684602251-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 14:51:57.195395 6982 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 14:51:57.198414 6982 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 14:51:57.198565 6982 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 14:51:57.205309 6982 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:37119
I20260927 14:51:57.205590 7049 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:37119 every 8 connection(s)
I20260927 14:51:57.206235 6982 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790520684602251-392-0/minicluster-data/master-1/data/info.pb
I20260927 14:51:57.208818 7050 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 14:51:57.212320 7050 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36527 } has no permanent_uuid. Determining permanent_uuid...
I20260927 14:51:57.212862 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6936m5/build/debug/bin/kudu as pid 6982
Sep 27 14:51:57 dist-test-slave-xk1g krb5kdc[6473](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790520717, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.61@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
Sep 27 14:51:57 dist-test-slave-xk1g krb5kdc[6473](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790520716, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260927 14:51:57.226049 6557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:51:57.214230 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55354 (local address 127.0.98.62:36527)
0927 14:51:57.214382 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0927 14:51:57.214387 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 14:51:57.214922 (+ 535us) server_negotiation.cc:407] Connection header received
0927 14:51:57.215631 (+ 709us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:51:57.215635 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:51:57.215681 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:51:57.215776 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:51:57.216950 (+ 1174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:51:57.217605 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:51:57.218616 (+ 1011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:51:57.218836 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:51:57.218948 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:51:57.221979 (+ 3031us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:51:57.221983 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:51:57.221987 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 14:51:57.222028 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:51:57.223904 (+ 1876us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:51:57.224395 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:51:57.224399 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:51:57.224402 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:51:57.224505 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:51:57.224882 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:51:57.224885 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:51:57.224888 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:51:57.225215 (+ 327us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:51:57.225322 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0927 14:51:57.225751 (+ 429us) server_negotiation.cc:299] Negotiation successful
0927 14:51:57.225883 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260927 14:51:57.226969 7053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:51:57.214305 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59872 (local address 127.0.98.61:37119)
0927 14:51:57.214689 (+ 384us) server_negotiation.cc:206] Beginning negotiation
0927 14:51:57.214711 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0927 14:51:57.214743 (+ 32us) server_negotiation.cc:407] Connection header received
0927 14:51:57.215410 (+ 667us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:51:57.215427 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:51:57.215651 (+ 224us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:51:57.215820 (+ 169us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:51:57.216685 (+ 865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:51:57.217696 (+ 1011us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:51:57.218640 (+ 944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:51:57.219046 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:51:57.219169 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:51:57.222399 (+ 3230us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:51:57.222409 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:51:57.222418 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 14:51:57.222437 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:51:57.224086 (+ 1649us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:51:57.224612 (+ 526us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:51:57.224617 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:51:57.224619 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:51:57.224772 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:51:57.225123 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:51:57.225125 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:51:57.225127 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:51:57.225510 (+ 383us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:51:57.225687 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0927 14:51:57.226005 (+ 318us) server_negotiation.cc:299] Negotiation successful
0927 14:51:57.226113 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":254,"spinlock_wait_cycles":1792,"thread_start_us":130,"threads_started":1}
I20260927 14:51:57.226970 7054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:51:57.214313 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36527 (local address 127.0.0.1:55354)
0927 14:51:57.214770 (+ 457us) negotiation.cc:107] Waiting for socket to connect
0927 14:51:57.214825 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0927 14:51:57.215431 (+ 606us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:51:57.216192 (+ 761us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:51:57.216196 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:51:57.216492 (+ 296us) client_negotiation.cc:189] Negotiated authn=SASL
0927 14:51:57.216794 (+ 302us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:51:57.216806 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:51:57.217739 (+ 933us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:51:57.217744 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:51:57.218486 (+ 742us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:51:57.218494 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:51:57.218989 (+ 495us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:51:57.218993 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:51:57.219174 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:51:57.219978 (+ 804us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 14:51:57.220002 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0927 14:51:57.221818 (+ 1816us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 14:51:57.224019 (+ 2201us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 14:51:57.224028 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 14:51:57.224038 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0927 14:51:57.224293 (+ 255us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 14:51:57.224605 (+ 312us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 14:51:57.224608 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 14:51:57.224610 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 14:51:57.224773 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 14:51:57.225334 (+ 561us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 14:51:57.225339 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 14:51:57.225509 (+ 170us) client_negotiation.cc:769] Sending connection context
0927 14:51:57.225674 (+ 165us) client_negotiation.cc:240] Negotiation successful
0927 14:51:57.225960 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":291,"spinlock_wait_cycles":54912,"thread_start_us":167,"threads_started":1}
I20260927 14:51:57.228323 7050 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37119 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 14:51:57 dist-test-slave-xk1g krb5kdc[6473](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790520717, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.61@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260927 14:51:57.235456 7054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:51:57.228766 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:37119 (local address 127.0.0.1:59886)
0927 14:51:57.228921 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0927 14:51:57.228937 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 14:51:57.229015 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:51:57.229283 (+ 268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:51:57.229285 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:51:57.229410 (+ 125us) client_negotiation.cc:189] Negotiated authn=SASL
0927 14:51:57.229490 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:51:57.229495 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:51:57.230125 (+ 630us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:51:57.230127 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:51:57.230612 (+ 485us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:51:57.230618 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:51:57.230981 (+ 363us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:51:57.230983 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:51:57.231046 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:51:57.231482 (+ 436us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 14:51:57.231492 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0927 14:51:57.232643 (+ 1151us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 14:51:57.234259 (+ 1616us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 14:51:57.234261 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 14:51:57.234262 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 14:51:57.234449 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 14:51:57.234717 (+ 268us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 14:51:57.234720 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 14:51:57.234721 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 14:51:57.234815 (+ 94us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 14:51:57.235124 (+ 309us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 14:51:57.235126 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 14:51:57.235189 (+ 63us) client_negotiation.cc:769] Sending connection context
0927 14:51:57.235264 (+ 75us) client_negotiation.cc:240] Negotiation successful
0927 14:51:57.235345 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":91}
I20260927 14:51:57.235638 7053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:51:57.228834 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59886 (local address 127.0.98.61:37119)
0927 14:51:57.228935 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0927 14:51:57.228938 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 14:51:57.228996 (+ 58us) server_negotiation.cc:407] Connection header received
0927 14:51:57.229134 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:51:57.229137 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:51:57.229182 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:51:57.229260 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:51:57.229582 (+ 322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:51:57.230029 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:51:57.230700 (+ 671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:51:57.230900 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:51:57.230970 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:51:57.232757 (+ 1787us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:51:57.232760 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:51:57.232761 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 14:51:57.232780 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:51:57.234156 (+ 1376us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:51:57.234526 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:51:57.234529 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:51:57.234530 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:51:57.234590 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:51:57.234891 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:51:57.234893 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:51:57.234895 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:51:57.235047 (+ 152us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:51:57.235119 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0927 14:51:57.235327 (+ 208us) server_negotiation.cc:299] Negotiation successful
0927 14:51:57.235406 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260927 14:51:57.238577 7050 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 413c118097b64cd8af3711ec05bf636a: Bootstrap starting.
I20260927 14:51:57.240659 7050 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 413c118097b64cd8af3711ec05bf636a: Neither blocks nor log segments found. Creating new log.
I20260927 14:51:57.241374 7050 log.cc:824] T 00000000000000000000000000000000 P 413c118097b64cd8af3711ec05bf636a: Log is configured to *not* fsync() on all Append() calls
I20260927 14:51:57.243126 7050 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 413c118097b64cd8af3711ec05bf636a: No bootstrap required, opened a new log
I20260927 14:51:57.245994 7050 raft_consensus.cc:348] T 00000000000000000000000000000000 P 413c118097b64cd8af3711ec05bf636a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d60b79f337e343bca73eadfa1bd55f6a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36527 } } peers { permanent_uuid: "413c118097b64cd8af3711ec05bf636a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37119 } }
I20260927 14:51:57.246173 7050 raft_consensus.cc:374] T 00000000000000000000000000000000 P 413c118097b64cd8af3711ec05bf636a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 14:51:57.246204 7050 raft_consensus.cc:729] T 00000000000000000000000000000000 P 413c118097b64cd8af3711ec05bf636a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 413c118097b64cd8af3711ec05bf636a, State: Initialized, Role: FOLLOWER
I20260927 14:51:57.246546 7050 consensus_queue.cc:260] T 00000000000000000000000000000000 P 413c118097b64cd8af3711ec05bf636a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d60b79f337e343bca73eadfa1bd55f6a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36527 } } peers { permanent_uuid: "413c118097b64cd8af3711ec05bf636a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37119 } }
I20260927 14:51:57.246923 7056 sys_catalog.cc:455] T 00000000000000000000000000000000 P 413c118097b64cd8af3711ec05bf636a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d60b79f337e343bca73eadfa1bd55f6a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36527 } } peers { permanent_uuid: "413c118097b64cd8af3711ec05bf636a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37119 } } }
I20260927 14:51:57.247090 7056 sys_catalog.cc:458] T 00000000000000000000000000000000 P 413c118097b64cd8af3711ec05bf636a [sys.catalog]: This master's current role is: FOLLOWER
I20260927 14:51:57.247597 7050 sys_catalog.cc:564] T 00000000000000000000000000000000 P 413c118097b64cd8af3711ec05bf636a [sys.catalog]: configured and running, proceeding with master startup.
W20260927 14:51:57.252102 7067 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 413c118097b64cd8af3711ec05bf636a: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 14:51:57.252180 7067 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 14:51:57.254033 6982 master_runner.cc:428] Detected that this master 413c118097b64cd8af3711ec05bf636a is joining an existing cluster
I20260927 14:51:57.254098 6982 master_runner.cc:432] Initiating AddMaster RPC to add 127.0.98.61:37119
I20260927 14:51:57.264485 7073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:51:57.256259 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36527 (local address 127.0.0.1:55368)
0927 14:51:57.256531 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0927 14:51:57.256546 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 14:51:57.256660 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:51:57.256944 (+ 284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:51:57.256947 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:51:57.257194 (+ 247us) client_negotiation.cc:189] Negotiated authn=SASL
0927 14:51:57.257372 (+ 178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:51:57.257379 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:51:57.258195 (+ 816us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:51:57.258200 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:51:57.258881 (+ 681us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:51:57.258889 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:51:57.259353 (+ 464us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:51:57.259356 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:51:57.259463 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:51:57.260042 (+ 579us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 14:51:57.260053 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 14:51:57.260373 (+ 320us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 14:51:57.262618 (+ 2245us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 14:51:57.262621 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 14:51:57.262623 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 14:51:57.262936 (+ 313us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 14:51:57.263250 (+ 314us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 14:51:57.263253 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 14:51:57.263255 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 14:51:57.263350 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 14:51:57.263876 (+ 526us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 14:51:57.263881 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 14:51:57.263988 (+ 107us) client_negotiation.cc:769] Sending connection context
0927 14:51:57.264130 (+ 142us) client_negotiation.cc:240] Negotiation successful
0927 14:51:57.264274 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":190,"mutex_wait_us":19,"thread_start_us":102,"threads_started":1}
I20260927 14:51:57.264493 6557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:51:57.256313 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55368 (local address 127.0.98.62:36527)
0927 14:51:57.256436 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0927 14:51:57.256441 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 14:51:57.256719 (+ 278us) server_negotiation.cc:407] Connection header received
0927 14:51:57.256771 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:51:57.256774 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:51:57.256826 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:51:57.256941 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:51:57.257491 (+ 550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:51:57.258094 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:51:57.258993 (+ 899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:51:57.259232 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:51:57.259345 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:51:57.260522 (+ 1177us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:51:57.260526 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:51:57.260535 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 14:51:57.260564 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:51:57.262277 (+ 1713us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:51:57.263047 (+ 770us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:51:57.263053 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:51:57.263056 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:51:57.263147 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:51:57.263466 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:51:57.263469 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:51:57.263471 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:51:57.263771 (+ 300us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:51:57.263870 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0927 14:51:57.264197 (+ 327us) server_negotiation.cc:299] Negotiation successful
0927 14:51:57.264325 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260927 14:51:57.264830 7053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:51:57.256578 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59896 (local address 127.0.98.61:37119)
0927 14:51:57.256685 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0927 14:51:57.256689 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 14:51:57.256975 (+ 286us) server_negotiation.cc:407] Connection header received
0927 14:51:57.257144 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:51:57.257147 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:51:57.257189 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:51:57.257266 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:51:57.257766 (+ 500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:51:57.258525 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:51:57.259714 (+ 1189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:51:57.260052 (+ 338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:51:57.260122 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:51:57.261593 (+ 1471us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:51:57.261598 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:51:57.261604 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 14:51:57.261633 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:51:57.263168 (+ 1535us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:51:57.263638 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:51:57.263641 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:51:57.263643 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:51:57.263711 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:51:57.263976 (+ 265us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:51:57.263981 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:51:57.263983 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:51:57.264178 (+ 195us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:51:57.264269 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0927 14:51:57.264562 (+ 293us) server_negotiation.cc:299] Negotiation successful
0927 14:51:57.264688 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":4,"server-negotiator.queue_time_us":41}
I20260927 14:51:57.265266 7074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:51:57.256259 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:37119 (local address 127.0.0.1:59896)
0927 14:51:57.256905 (+ 646us) negotiation.cc:107] Waiting for socket to connect
0927 14:51:57.256918 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 14:51:57.257022 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:51:57.257305 (+ 283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:51:57.257308 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:51:57.257531 (+ 223us) client_negotiation.cc:189] Negotiated authn=SASL
0927 14:51:57.257649 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:51:57.257656 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:51:57.258717 (+ 1061us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:51:57.258720 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:51:57.259577 (+ 857us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:51:57.259587 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:51:57.260278 (+ 691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:51:57.260281 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:51:57.260368 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:51:57.260805 (+ 437us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 14:51:57.260819 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0927 14:51:57.261252 (+ 433us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 14:51:57.263271 (+ 2019us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 14:51:57.263276 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 14:51:57.263278 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 14:51:57.263494 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 14:51:57.263811 (+ 317us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 14:51:57.263815 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 14:51:57.263816 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 14:51:57.263896 (+ 80us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 14:51:57.264304 (+ 408us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 14:51:57.264307 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 14:51:57.264381 (+ 74us) client_negotiation.cc:769] Sending connection context
0927 14:51:57.264938 (+ 557us) client_negotiation.cc:240] Negotiation successful
0927 14:51:57.265102 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":566,"mutex_wait_us":373,"thread_start_us":96,"threads_started":1}
Sep 27 14:51:57 dist-test-slave-xk1g krb5kdc[6473](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790520716, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260927 14:51:57.278090 7053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:51:57.269168 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59902 (local address 127.0.98.61:37119)
0927 14:51:57.269283 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0927 14:51:57.269290 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 14:51:57.269712 (+ 422us) server_negotiation.cc:407] Connection header received
0927 14:51:57.269878 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:51:57.269880 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:51:57.269920 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:51:57.269991 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:51:57.270765 (+ 774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:51:57.271243 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:51:57.272444 (+ 1201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:51:57.272636 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:51:57.272710 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:51:57.274978 (+ 2268us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:51:57.274981 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:51:57.274983 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 14:51:57.275002 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:51:57.276346 (+ 1344us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:51:57.276804 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:51:57.276809 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:51:57.276811 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:51:57.276886 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:51:57.277223 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:51:57.277226 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:51:57.277227 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:51:57.277384 (+ 157us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:51:57.277489 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0927 14:51:57.277817 (+ 328us) server_negotiation.cc:299] Negotiation successful
0927 14:51:57.277924 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260927 14:51:57.278092 7075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:51:57.269166 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:37119 (local address 127.0.0.1:59902)
0927 14:51:57.269569 (+ 403us) negotiation.cc:107] Waiting for socket to connect
0927 14:51:57.269627 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0927 14:51:57.269759 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:51:57.270047 (+ 288us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:51:57.270053 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:51:57.270521 (+ 468us) client_negotiation.cc:189] Negotiated authn=SASL
0927 14:51:57.270642 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:51:57.270649 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:51:57.271344 (+ 695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:51:57.271348 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:51:57.272334 (+ 986us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:51:57.272344 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:51:57.272740 (+ 396us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:51:57.272744 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:51:57.272850 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:51:57.273350 (+ 500us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 14:51:57.273364 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0927 14:51:57.274829 (+ 1465us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 14:51:57.276462 (+ 1633us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 14:51:57.276472 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 14:51:57.276478 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0927 14:51:57.276704 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 14:51:57.276986 (+ 282us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 14:51:57.276990 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 14:51:57.276992 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 14:51:57.277122 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 14:51:57.277503 (+ 381us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 14:51:57.277511 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 14:51:57.277624 (+ 113us) client_negotiation.cc:769] Sending connection context
0927 14:51:57.277757 (+ 133us) client_negotiation.cc:240] Negotiation successful
0927 14:51:57.277936 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":207,"thread_start_us":103,"threads_started":1}
I20260927 14:51:57.279371 6518 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.0.98.61:37119: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "413c118097b64cd8af3711ec05bf636a"
member_type: NON_VOTER
last_known_addr {
host: "127.0.98.61"
port: 37119
}
attrs {
promote: true
}
}
dest_uuid: "d60b79f337e343bca73eadfa1bd55f6a"
cas_config_opid_index: -1
I20260927 14:51:57.280087 6518 consensus_queue.cc:237] T 00000000000000000000000000000000 P d60b79f337e343bca73eadfa1bd55f6a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "d60b79f337e343bca73eadfa1bd55f6a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36527 } } peers { permanent_uuid: "413c118097b64cd8af3711ec05bf636a" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 37119 } attrs { promote: true } }
I20260927 14:51:57.282025 7076 raft_consensus.cc:2932] T 00000000000000000000000000000000 P d60b79f337e343bca73eadfa1bd55f6a [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 413c118097b64cd8af3711ec05bf636a (127.0.98.61) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "d60b79f337e343bca73eadfa1bd55f6a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36527 } } peers { permanent_uuid: "413c118097b64cd8af3711ec05bf636a" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 37119 } attrs { promote: true } } }
I20260927 14:51:57.282531 7024 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 413c118097b64cd8af3711ec05bf636a [term 0 FOLLOWER]: Advancing to term 1
I20260927 14:51:57.283702 7076 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.0.98.61:37119
I20260927 14:51:57.283895 7076 sys_catalog.cc:455] T 00000000000000000000000000000000 P d60b79f337e343bca73eadfa1bd55f6a [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "d60b79f337e343bca73eadfa1bd55f6a" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "d60b79f337e343bca73eadfa1bd55f6a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36527 } } peers { permanent_uuid: "413c118097b64cd8af3711ec05bf636a" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 37119 } attrs { promote: true } } }
I20260927 14:51:57.283980 7076 sys_catalog.cc:458] T 00000000000000000000000000000000 P d60b79f337e343bca73eadfa1bd55f6a [sys.catalog]: This master's current role is: LEADER
I20260927 14:51:57.284016 7024 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 413c118097b64cd8af3711ec05bf636a [term 1 FOLLOWER]: Refusing update from remote peer d60b79f337e343bca73eadfa1bd55f6a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260927 14:51:57.284106 6982 master.cc:537] Master@127.0.98.61:37119 shutting down...
I20260927 14:51:57.285113 7077 consensus_queue.cc:1035] T 00000000000000000000000000000000 P d60b79f337e343bca73eadfa1bd55f6a [LEADER]: Connected to new peer: Peer: permanent_uuid: "413c118097b64cd8af3711ec05bf636a" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 37119 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 14:51:57.288496 6982 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 413c118097b64cd8af3711ec05bf636a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 14:51:57.288761 6982 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 413c118097b64cd8af3711ec05bf636a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 14:51:57.288865 6982 tablet_replica.cc:330] T 00000000000000000000000000000000 P 413c118097b64cd8af3711ec05bf636a: stopping tablet replica
W20260927 14:51:57.289039 6504 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
WARNING: no policy specified for oryx/127.0.98.60@KRBTEST.COM; defaulting to no policy
W20260927 14:51:57.291303 6504 consensus_peers.cc:487] T 00000000000000000000000000000000 P d60b79f337e343bca73eadfa1bd55f6a -> Peer 413c118097b64cd8af3711ec05bf636a (127.0.98.61:37119): Couldn't send request to peer 413c118097b64cd8af3711ec05bf636a. Status: Network error: recv got EOF from unknown peer (error 108). This is attempt 1: this message will repeat every 5th retry.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.60@KRBTEST.COM" created.
I20260927 14:51:57.303911 6982 master.cc:552] Master@127.0.98.61:37119 shutdown complete.
I20260927 14:51:57.303969 6982 master_runner.cc:305] Clearing existing system tablet
I20260927 14:51:57.304207 6982 fs_manager.cc:359] Metadata directory not provided
I20260927 14:51:57.304250 6982 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 14:51:57.305555 6982 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.002s
I20260927 14:51:57.306425 7084 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 14:51:57.306718 6982 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790520684602251-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 14:51:57.306861 6982 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260927 14:51:57.306903 6982 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790520684602251-392-0/minicluster-data/master-1/data,/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790520684602251-392-0/minicluster-data/master-1/wal
uuid: "413c118097b64cd8af3711ec05bf636a"
format_stamp: "Formatted at 2026-09-27 14:51:57 on dist-test-slave-xk1g"
server_key: "0ff963f6b0c0f7368489baa55ad85b67"
server_key_iv: "8538639a3832603e4d21c174afa7eeeb"
server_key_version: "encryptionkey@0"
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790520684602251-392-0/krb5kdc/oryx_127.0.98.60.keytab.
Entry for principal oryx/127.0.98.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790520684602251-392-0/krb5kdc/oryx_127.0.98.60.keytab.
I20260927 14:51:57.308403 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6936m5/build/debug/bin/kudu
/tmp/dist-test-task6936m5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790520684602251-392-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790520684602251-392-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790520684602251-392-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790520684602251-392-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:38741
--webserver_interface=127.0.98.60
--webserver_port=0
--keytab_file=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790520684602251-392-0/krb5kdc/oryx_127.0.98.60.keytab
--principal=oryx/127.0.98.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44229
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:36527,127.0.98.61:37119,127.0.98.60:38741
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790520684602251-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790520684602251-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790520684602251-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 14:51:57.308946 6982 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 413c118097b64cd8af3711ec05bf636a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 14:51:57.311133 6982 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 413c118097b64cd8af3711ec05bf636a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260927 14:51:57.311231 6982 log.cc:1196] T 00000000000000000000000000000000 P 413c118097b64cd8af3711ec05bf636a: Deleting WAL directory at /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790520684602251-392-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260927 14:51:57.311499 6982 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 413c118097b64cd8af3711ec05bf636a: Deleting consensus metadata
I20260927 14:51:57.311671 6982 master_runner.cc:315] Copying system tablet from 127.0.98.62:36527
I20260927 14:51:57.313079 6982 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 413c118097b64cd8af3711ec05bf636a: tablet copy: Beginning tablet copy session from remote peer at address 127.0.98.62:36527
I20260927 14:51:57.326716 6557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:51:57.317600 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40352 (local address 127.0.98.62:36527)
0927 14:51:57.317720 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0927 14:51:57.317725 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 14:51:57.318232 (+ 507us) server_negotiation.cc:407] Connection header received
0927 14:51:57.318472 (+ 240us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:51:57.318478 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:51:57.318537 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:51:57.318597 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:51:57.319352 (+ 755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:51:57.320229 (+ 877us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:51:57.321436 (+ 1207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:51:57.321765 (+ 329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:51:57.321832 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:51:57.323235 (+ 1403us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:51:57.323239 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:51:57.323242 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 14:51:57.323265 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:51:57.324954 (+ 1689us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:51:57.325474 (+ 520us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:51:57.325477 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:51:57.325479 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:51:57.325564 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:51:57.325842 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:51:57.325845 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:51:57.325846 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:51:57.326045 (+ 199us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:51:57.326112 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0927 14:51:57.326449 (+ 337us) server_negotiation.cc:299] Negotiation successful
0927 14:51:57.326561 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260927 14:51:57.327262 7091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:51:57.317744 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36527 (local address 127.0.0.1:40352)
0927 14:51:57.318132 (+ 388us) negotiation.cc:107] Waiting for socket to connect
0927 14:51:57.318148 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 14:51:57.318339 (+ 191us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:51:57.318773 (+ 434us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:51:57.318779 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:51:57.319020 (+ 241us) client_negotiation.cc:189] Negotiated authn=SASL
0927 14:51:57.319176 (+ 156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:51:57.319189 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:51:57.320384 (+ 1195us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:51:57.320388 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:51:57.321300 (+ 912us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:51:57.321318 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:51:57.321980 (+ 662us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:51:57.321983 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:51:57.322090 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:51:57.322661 (+ 571us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 14:51:57.322676 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 14:51:57.323080 (+ 404us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 14:51:57.325102 (+ 2022us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 14:51:57.325106 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 14:51:57.325108 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 14:51:57.325366 (+ 258us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 14:51:57.325665 (+ 299us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 14:51:57.325668 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 14:51:57.325670 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 14:51:57.325746 (+ 76us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 14:51:57.326180 (+ 434us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 14:51:57.326183 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 14:51:57.326269 (+ 86us) client_negotiation.cc:769] Sending connection context
0927 14:51:57.326928 (+ 659us) client_negotiation.cc:240] Negotiation successful
0927 14:51:57.327073 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":292,"thread_start_us":100,"threads_started":1}
I20260927 14:51:57.327766 6538 tablet_copy_service.cc:140] P d60b79f337e343bca73eadfa1bd55f6a: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 413c118097b64cd8af3711ec05bf636a ({username='oryx', principal='oryx/127.0.98.61@KRBTEST.COM'} at 127.0.0.1:40352)
I20260927 14:51:57.327943 6538 tablet_copy_service.cc:161] P d60b79f337e343bca73eadfa1bd55f6a: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 413c118097b64cd8af3711ec05bf636a at {username='oryx', principal='oryx/127.0.98.61@KRBTEST.COM'} at 127.0.0.1:40352: session id = 413c118097b64cd8af3711ec05bf636a-00000000000000000000000000000000
I20260927 14:51:57.329732 6538 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P d60b79f337e343bca73eadfa1bd55f6a: Tablet Copy: opened 0 blocks and 1 log segments
I20260927 14:51:57.331629 6982 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260927 14:51:57.334084 6982 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 413c118097b64cd8af3711ec05bf636a: tablet copy: Starting download of 0 data blocks...
I20260927 14:51:57.334267 6982 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 413c118097b64cd8af3711ec05bf636a: tablet copy: Starting download of 1 WAL segments...
I20260927 14:51:57.336092 6982 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 413c118097b64cd8af3711ec05bf636a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260927 14:51:57.337901 6538 tablet_copy_service.cc:342] P d60b79f337e343bca73eadfa1bd55f6a: Request end of tablet copy session 413c118097b64cd8af3711ec05bf636a-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.0.98.61@KRBTEST.COM'} at 127.0.0.1:40352
I20260927 14:51:57.338080 6538 tablet_copy_service.cc:433] P d60b79f337e343bca73eadfa1bd55f6a: ending tablet copy session 413c118097b64cd8af3711ec05bf636a-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 413c118097b64cd8af3711ec05bf636a
I20260927 14:51:57.345999 6982 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 14:51:57.347574 7092 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 14:51:57.349978 7093 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 14:51:57.350236 7095 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 14:51:57.350464 6982 server_base.cc:1034] running on GCE node
I20260927 14:51:57.350610 6982 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 14:51:57.350880 6982 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 14:51:57.352006 6982 hybrid_clock.cc:648] HybridClock initialized: now 1790520717351993 us; error 29 us; skew 500 ppm
Sep 27 14:51:57 dist-test-slave-xk1g krb5kdc[6473](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790520717, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 14:51:57.353631 6982 init.cc:377] Logged in from keytab as oryx/127.0.98.61@KRBTEST.COM (short username oryx)
I20260927 14:51:57.354195 6982 webserver.cc:460] Webserver started at http://127.0.98.61:34597/ using document root <none> and password file <none>
I20260927 14:51:57.354396 6982 fs_manager.cc:359] Metadata directory not provided
I20260927 14:51:57.354439 6982 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 14:51:57.355872 6982 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260927 14:51:57.356841 7102 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 14:51:57.357154 6982 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 14:51:57.357234 6982 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790520684602251-392-0/minicluster-data/master-1/data,/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790520684602251-392-0/minicluster-data/master-1/wal
uuid: "413c118097b64cd8af3711ec05bf636a"
format_stamp: "Formatted at 2026-09-27 14:51:57 on dist-test-slave-xk1g"
server_key: "0ff963f6b0c0f7368489baa55ad85b67"
server_key_iv: "8538639a3832603e4d21c174afa7eeeb"
server_key_version: "encryptionkey@0"
I20260927 14:51:57.357326 6982 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790520684602251-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790520684602251-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790520684602251-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 14:51:57.370033 6982 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 14:51:57.372493 6982 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 14:51:57.377277 6982 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:37119
I20260927 14:51:57.377307 7154 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:37119 every 8 connection(s)
I20260927 14:51:57.377874 6982 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790520684602251-392-0/minicluster-data/master-1/data/info.pb
I20260927 14:51:57.379680 7155 sys_catalog.cc:263] Verifying existing consensus state
I20260927 14:51:57.380692 7155 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 413c118097b64cd8af3711ec05bf636a: Bootstrap starting.
I20260927 14:51:57.391059 7155 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 413c118097b64cd8af3711ec05bf636a: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 14:51:57.391377 7155 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 413c118097b64cd8af3711ec05bf636a: Bootstrap complete.
I20260927 14:51:57.391960 7155 raft_consensus.cc:348] T 00000000000000000000000000000000 P 413c118097b64cd8af3711ec05bf636a [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "d60b79f337e343bca73eadfa1bd55f6a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36527 } } peers { permanent_uuid: "413c118097b64cd8af3711ec05bf636a" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 37119 } attrs { promote: true } }
I20260927 14:51:57.392097 7155 raft_consensus.cc:729] T 00000000000000000000000000000000 P 413c118097b64cd8af3711ec05bf636a [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 413c118097b64cd8af3711ec05bf636a, State: Initialized, Role: LEARNER
I20260927 14:51:57.392213 7155 consensus_queue.cc:260] T 00000000000000000000000000000000 P 413c118097b64cd8af3711ec05bf636a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "d60b79f337e343bca73eadfa1bd55f6a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36527 } } peers { permanent_uuid: "413c118097b64cd8af3711ec05bf636a" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 37119 } attrs { promote: true } }
I20260927 14:51:57.392571 7158 sys_catalog.cc:455] T 00000000000000000000000000000000 P 413c118097b64cd8af3711ec05bf636a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "d60b79f337e343bca73eadfa1bd55f6a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36527 } } peers { permanent_uuid: "413c118097b64cd8af3711ec05bf636a" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 37119 } attrs { promote: true } } }
I20260927 14:51:57.392676 7155 sys_catalog.cc:564] T 00000000000000000000000000000000 P 413c118097b64cd8af3711ec05bf636a [sys.catalog]: configured and running, proceeding with master startup.
I20260927 14:51:57.392755 7158 sys_catalog.cc:458] T 00000000000000000000000000000000 P 413c118097b64cd8af3711ec05bf636a [sys.catalog]: This master's current role is: LEARNER
I20260927 14:51:57.395933 7169 catalog_manager.cc:1259] Loaded cluster ID: 3ea1e4dfc1684ee8ba96f40d83c0eb15
I20260927 14:51:57.396044 7169 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 413c118097b64cd8af3711ec05bf636a: loading cluster ID for follower catalog manager: success
I20260927 14:51:57.398159 7169 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 413c118097b64cd8af3711ec05bf636a: acquiring CA information for follower catalog manager: success
I20260927 14:51:57.399482 7169 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 413c118097b64cd8af3711ec05bf636a: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260927 14:51:57.440546 7085 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 14:51:57.440788 7085 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 14:51:57.440868 7085 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 14:51:57.445035 7085 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 14:51:57.445102 7085 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 14:51:57.445122 7085 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 14:51:57.445138 7085 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 14:51:57.445153 7085 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 14:51:57.450385 7085 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44229
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790520684602251-392-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790520684602251-392-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:36527,127.0.98.61:37119,127.0.98.60:38741
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790520684602251-392-0/krb5kdc/oryx_127.0.98.60.keytab
--principal=oryx/127.0.98.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:38741
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790520684602251-392-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.98.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7085
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790520684602251-392-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:15:17 UTC on ccycloud.quasar-ledxfe.root.comops.site
build id 4676134
I20260927 14:51:57.451457 7085 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 14:51:57.452524 7085 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 14:51:57.460902 7174 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 14:51:57.460855 7177 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 14:51:57.461087 7085 server_base.cc:1034] running on GCE node
W20260927 14:51:57.461112 7175 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 14:51:57.461853 7085 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 14:51:57.462875 7085 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 14:51:57.464031 7085 hybrid_clock.cc:648] HybridClock initialized: now 1790520717464014 us; error 28 us; skew 500 ppm
Sep 27 14:51:57 dist-test-slave-xk1g krb5kdc[6473](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790520717, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 14:51:57.466763 7085 init.cc:377] Logged in from keytab as oryx/127.0.98.60@KRBTEST.COM (short username oryx)
I20260927 14:51:57.467571 7085 webserver.cc:460] Webserver started at http://127.0.98.60:46237/ using document root <none> and password file <none>
I20260927 14:51:57.468107 7085 fs_manager.cc:359] Metadata directory not provided
I20260927 14:51:57.468194 7085 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 14:51:57.468426 7085 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 14:51:57.470384 7085 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790520684602251-392-0/minicluster-data/master-2/data/instance:
uuid: "34720d7704db415f8c02e5d750103c4a"
format_stamp: "Formatted at 2026-09-27 14:51:57 on dist-test-slave-xk1g"
server_key: "5a3ba0183b88af4df2206b6556f1cc46"
server_key_iv: "78c99eb03462845affb95889baf526c4"
server_key_version: "encryptionkey@0"
I20260927 14:51:57.470938 7085 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790520684602251-392-0/minicluster-data/master-2/wal/instance:
uuid: "34720d7704db415f8c02e5d750103c4a"
format_stamp: "Formatted at 2026-09-27 14:51:57 on dist-test-slave-xk1g"
server_key: "5a3ba0183b88af4df2206b6556f1cc46"
server_key_iv: "78c99eb03462845affb95889baf526c4"
server_key_version: "encryptionkey@0"
I20260927 14:51:57.474511 7085 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.004s
I20260927 14:51:57.476765 7184 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 14:51:57.477946 7085 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 14:51:57.478084 7085 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790520684602251-392-0/minicluster-data/master-2/data,/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790520684602251-392-0/minicluster-data/master-2/wal
uuid: "34720d7704db415f8c02e5d750103c4a"
format_stamp: "Formatted at 2026-09-27 14:51:57 on dist-test-slave-xk1g"
server_key: "5a3ba0183b88af4df2206b6556f1cc46"
server_key_iv: "78c99eb03462845affb95889baf526c4"
server_key_version: "encryptionkey@0"
I20260927 14:51:57.478191 7085 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790520684602251-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790520684602251-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790520684602251-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 14:51:57.499920 7085 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 14:51:57.503080 7085 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 14:51:57.503278 7085 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 14:51:57.509999 7085 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:38741
I20260927 14:51:57.510269 7236 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:38741 every 8 connection(s)
I20260927 14:51:57.510954 7085 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790520684602251-392-0/minicluster-data/master-2/data/info.pb
I20260927 14:51:57.513656 7237 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 14:51:57.515441 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6936m5/build/debug/bin/kudu as pid 7085
I20260927 14:51:57.517489 7237 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36527 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 14:51:57 dist-test-slave-xk1g krb5kdc[6473](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790520716, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
Sep 27 14:51:57 dist-test-slave-xk1g krb5kdc[6473](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790520717, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260927 14:51:57.529332 6557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:51:57.519595 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40364 (local address 127.0.98.62:36527)
0927 14:51:57.519718 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0927 14:51:57.519723 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 14:51:57.519919 (+ 196us) server_negotiation.cc:407] Connection header received
0927 14:51:57.520082 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:51:57.520085 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:51:57.520137 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:51:57.520221 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:51:57.520931 (+ 710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:51:57.521492 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:51:57.522579 (+ 1087us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:51:57.522907 (+ 328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:51:57.523036 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:51:57.526019 (+ 2983us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:51:57.526023 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:51:57.526025 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 14:51:57.526057 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:51:57.527505 (+ 1448us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:51:57.528043 (+ 538us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:51:57.528048 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:51:57.528052 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:51:57.528149 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:51:57.528404 (+ 255us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:51:57.528406 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:51:57.528407 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:51:57.528557 (+ 150us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:51:57.528663 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0927 14:51:57.529092 (+ 429us) server_negotiation.cc:299] Negotiation successful
0927 14:51:57.529212 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260927 14:51:57.530202 7240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:51:57.519431 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36527 (local address 127.0.0.1:40364)
0927 14:51:57.519801 (+ 370us) negotiation.cc:107] Waiting for socket to connect
0927 14:51:57.519844 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0927 14:51:57.519970 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:51:57.520265 (+ 295us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:51:57.520270 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:51:57.520598 (+ 328us) client_negotiation.cc:189] Negotiated authn=SASL
0927 14:51:57.520791 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:51:57.520800 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:51:57.521620 (+ 820us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:51:57.521623 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:51:57.522449 (+ 826us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:51:57.522462 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:51:57.523066 (+ 604us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:51:57.523070 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:51:57.523179 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:51:57.523784 (+ 605us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 14:51:57.523807 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0927 14:51:57.525858 (+ 2051us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 14:51:57.527604 (+ 1746us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 14:51:57.527610 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 14:51:57.527619 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0927 14:51:57.527944 (+ 325us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 14:51:57.528229 (+ 285us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 14:51:57.528232 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 14:51:57.528234 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 14:51:57.528318 (+ 84us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 14:51:57.528672 (+ 354us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 14:51:57.528677 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 14:51:57.528844 (+ 167us) client_negotiation.cc:769] Sending connection context
0927 14:51:57.528988 (+ 144us) client_negotiation.cc:240] Negotiation successful
0927 14:51:57.529280 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":213,"mutex_wait_us":97,"spinlock_wait_cycles":19328,"thread_start_us":110,"threads_started":1}
I20260927 14:51:57.530202 7239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:51:57.516841 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44596 (local address 127.0.98.60:38741)
0927 14:51:57.517267 (+ 426us) server_negotiation.cc:206] Beginning negotiation
0927 14:51:57.517284 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0927 14:51:57.517320 (+ 36us) server_negotiation.cc:407] Connection header received
0927 14:51:57.517927 (+ 607us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:51:57.517942 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:51:57.518297 (+ 355us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:51:57.518626 (+ 329us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:51:57.519422 (+ 796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:51:57.520593 (+ 1171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:51:57.521597 (+ 1004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:51:57.522067 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:51:57.522182 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:51:57.525221 (+ 3039us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:51:57.525226 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:51:57.525235 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 14:51:57.525274 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:51:57.527111 (+ 1837us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:51:57.527610 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:51:57.527615 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:51:57.527617 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:51:57.527784 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:51:57.528088 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:51:57.528090 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:51:57.528091 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:51:57.528545 (+ 454us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:51:57.528715 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0927 14:51:57.528932 (+ 217us) server_negotiation.cc:299] Negotiation successful
0927 14:51:57.529230 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":294,"thread_start_us":142,"threads_started":1}
I20260927 14:51:57.531706 7237 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37119 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 14:51:57 dist-test-slave-xk1g krb5kdc[6473](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790520717, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260927 14:51:57.539368 7240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:51:57.532196 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:37119 (local address 127.0.0.1:52178)
0927 14:51:57.532295 (+ 99us) negotiation.cc:107] Waiting for socket to connect
0927 14:51:57.532308 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 14:51:57.532375 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:51:57.532776 (+ 401us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:51:57.532779 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:51:57.532942 (+ 163us) client_negotiation.cc:189] Negotiated authn=SASL
0927 14:51:57.533021 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:51:57.533025 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:51:57.533917 (+ 892us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:51:57.533922 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:51:57.534449 (+ 527us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:51:57.534457 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:51:57.534931 (+ 474us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:51:57.534934 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:51:57.535029 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:51:57.535470 (+ 441us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 14:51:57.535483 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0927 14:51:57.536702 (+ 1219us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 14:51:57.538241 (+ 1539us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 14:51:57.538243 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 14:51:57.538245 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 14:51:57.538426 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 14:51:57.538646 (+ 220us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 14:51:57.538648 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 14:51:57.538649 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 14:51:57.538714 (+ 65us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 14:51:57.539017 (+ 303us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 14:51:57.539019 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 14:51:57.539086 (+ 67us) client_negotiation.cc:769] Sending connection context
0927 14:51:57.539161 (+ 75us) client_negotiation.cc:240] Negotiation successful
0927 14:51:57.539246 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":27}
I20260927 14:51:57.539415 7241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:51:57.532361 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52178 (local address 127.0.98.61:37119)
0927 14:51:57.532587 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0927 14:51:57.532590 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 14:51:57.532601 (+ 11us) server_negotiation.cc:407] Connection header received
0927 14:51:57.532634 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:51:57.532636 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:51:57.532670 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:51:57.532764 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:51:57.533158 (+ 394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:51:57.533810 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:51:57.534565 (+ 755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:51:57.534824 (+ 259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:51:57.534933 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:51:57.536821 (+ 1888us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:51:57.536824 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:51:57.536825 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 14:51:57.536849 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:51:57.538145 (+ 1296us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:51:57.538504 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:51:57.538506 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:51:57.538508 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:51:57.538573 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:51:57.538784 (+ 211us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:51:57.538786 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:51:57.538787 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:51:57.538936 (+ 149us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:51:57.539016 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0927 14:51:57.539211 (+ 195us) server_negotiation.cc:299] Negotiation successful
0927 14:51:57.539301 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":166,"thread_start_us":74,"threads_started":1}
I20260927 14:51:57.540123 7237 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38741 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 14:51:57 dist-test-slave-xk1g krb5kdc[6473](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790520717, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
I20260927 14:51:57.547900 7240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:51:57.540558 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:38741 (local address 127.0.0.1:44610)
0927 14:51:57.540663 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0927 14:51:57.540676 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 14:51:57.540753 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:51:57.540984 (+ 231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:51:57.540987 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:51:57.541160 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0927 14:51:57.541215 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:51:57.541219 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:51:57.542230 (+ 1011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:51:57.542236 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:51:57.542900 (+ 664us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:51:57.542909 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:51:57.543362 (+ 453us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:51:57.543366 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:51:57.543473 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:51:57.543959 (+ 486us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 14:51:57.543972 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0927 14:51:57.545106 (+ 1134us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 14:51:57.546733 (+ 1627us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 14:51:57.546735 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 14:51:57.546737 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 14:51:57.546916 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 14:51:57.547145 (+ 229us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 14:51:57.547147 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 14:51:57.547148 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 14:51:57.547218 (+ 70us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 14:51:57.547555 (+ 337us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 14:51:57.547557 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 14:51:57.547623 (+ 66us) client_negotiation.cc:769] Sending connection context
0927 14:51:57.547698 (+ 75us) client_negotiation.cc:240] Negotiation successful
0927 14:51:57.547791 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":25,"mutex_wait_us":20}
I20260927 14:51:57.548059 7239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:51:57.540594 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44610 (local address 127.0.98.60:38741)
0927 14:51:57.540692 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0927 14:51:57.540698 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 14:51:57.540711 (+ 13us) server_negotiation.cc:407] Connection header received
0927 14:51:57.540860 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:51:57.540862 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:51:57.540901 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:51:57.540975 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:51:57.541313 (+ 338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:51:57.542122 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:51:57.543010 (+ 888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:51:57.543248 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:51:57.543366 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:51:57.545228 (+ 1862us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:51:57.545231 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:51:57.545233 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 14:51:57.545253 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:51:57.546646 (+ 1393us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:51:57.546998 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:51:57.547000 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:51:57.547002 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:51:57.547069 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:51:57.547291 (+ 222us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:51:57.547293 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:51:57.547294 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:51:57.547477 (+ 183us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:51:57.547552 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0927 14:51:57.547727 (+ 175us) server_negotiation.cc:299] Negotiation successful
0927 14:51:57.547859 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":45,"server-negotiator.queue_time_us":43}
I20260927 14:51:57.550702 7237 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 34720d7704db415f8c02e5d750103c4a: Bootstrap starting.
I20260927 14:51:57.552964 7237 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 34720d7704db415f8c02e5d750103c4a: Neither blocks nor log segments found. Creating new log.
I20260927 14:51:57.553768 7237 log.cc:824] T 00000000000000000000000000000000 P 34720d7704db415f8c02e5d750103c4a: Log is configured to *not* fsync() on all Append() calls
I20260927 14:51:57.555501 7237 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 34720d7704db415f8c02e5d750103c4a: No bootstrap required, opened a new log
I20260927 14:51:57.558550 7237 raft_consensus.cc:348] T 00000000000000000000000000000000 P 34720d7704db415f8c02e5d750103c4a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d60b79f337e343bca73eadfa1bd55f6a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36527 } } peers { permanent_uuid: "413c118097b64cd8af3711ec05bf636a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37119 } } peers { permanent_uuid: "34720d7704db415f8c02e5d750103c4a" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38741 } }
I20260927 14:51:57.558777 7237 raft_consensus.cc:374] T 00000000000000000000000000000000 P 34720d7704db415f8c02e5d750103c4a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 14:51:57.558851 7237 raft_consensus.cc:729] T 00000000000000000000000000000000 P 34720d7704db415f8c02e5d750103c4a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 34720d7704db415f8c02e5d750103c4a, State: Initialized, Role: FOLLOWER
I20260927 14:51:57.559279 7237 consensus_queue.cc:260] T 00000000000000000000000000000000 P 34720d7704db415f8c02e5d750103c4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d60b79f337e343bca73eadfa1bd55f6a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36527 } } peers { permanent_uuid: "413c118097b64cd8af3711ec05bf636a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37119 } } peers { permanent_uuid: "34720d7704db415f8c02e5d750103c4a" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38741 } }
I20260927 14:51:57.559777 7243 sys_catalog.cc:455] T 00000000000000000000000000000000 P 34720d7704db415f8c02e5d750103c4a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d60b79f337e343bca73eadfa1bd55f6a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36527 } } peers { permanent_uuid: "413c118097b64cd8af3711ec05bf636a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37119 } } peers { permanent_uuid: "34720d7704db415f8c02e5d750103c4a" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38741 } } }
I20260927 14:51:57.559906 7243 sys_catalog.cc:458] T 00000000000000000000000000000000 P 34720d7704db415f8c02e5d750103c4a [sys.catalog]: This master's current role is: FOLLOWER
I20260927 14:51:57.560503 7237 sys_catalog.cc:564] T 00000000000000000000000000000000 P 34720d7704db415f8c02e5d750103c4a [sys.catalog]: configured and running, proceeding with master startup.
W20260927 14:51:57.565429 7254 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 34720d7704db415f8c02e5d750103c4a: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 14:51:57.565508 7254 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 14:51:57.568857 7085 master_runner.cc:428] Detected that this master 34720d7704db415f8c02e5d750103c4a is joining an existing cluster
I20260927 14:51:57.568943 7085 master_runner.cc:432] Initiating AddMaster RPC to add 127.0.98.60:38741
I20260927 14:51:57.583597 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6936m5/build/debug/bin/kudu with pid 6580
I20260927 14:51:57.590042 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6936m5/build/debug/bin/kudu
/tmp/dist-test-task6936m5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790520684602251-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790520684602251-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790520684602251-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790520684602251-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:43017
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:36527
--webserver_port=33431
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790520684602251-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44229
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.98.62:36527,127.0.98.61:37119
--tserver_master_addrs=127.0.98.62:36527,127.0.98.61:37119,127.0.98.60:38741 with env {KRB5CCNAME=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790520684602251-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790520684602251-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790520684602251-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 14:51:57.599164 7261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:51:57.571735 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:38741 (local address 127.0.0.1:44616)
0927 14:51:57.573524 (+ 1789us) negotiation.cc:107] Waiting for socket to connect
0927 14:51:57.573537 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 14:51:57.574078 (+ 541us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:51:57.574429 (+ 351us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:51:57.574432 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:51:57.574634 (+ 202us) client_negotiation.cc:189] Negotiated authn=SASL
0927 14:51:57.575008 (+ 374us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:51:57.575017 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:51:57.575959 (+ 942us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:51:57.575962 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:51:57.576837 (+ 875us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:51:57.576847 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:51:57.577395 (+ 548us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:51:57.577399 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:51:57.577523 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:51:57.578097 (+ 574us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 14:51:57.578120 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0927 14:51:57.578570 (+ 450us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 14:51:57.580328 (+ 1758us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 14:51:57.580335 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 14:51:57.580338 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 14:51:57.597761 (+ 17423us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 14:51:57.598095 (+ 334us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 14:51:57.598101 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 14:51:57.598103 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 14:51:57.598191 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 14:51:57.598627 (+ 436us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 14:51:57.598631 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 14:51:57.598715 (+ 84us) client_negotiation.cc:769] Sending connection context
0927 14:51:57.598817 (+ 102us) client_negotiation.cc:240] Negotiation successful
0927 14:51:57.598971 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1717,"mutex_wait_us":24,"thread_start_us":104,"threads_started":1}
I20260927 14:51:57.599224 6557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:51:57.571087 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40372 (local address 127.0.98.62:36527)
0927 14:51:57.571296 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0927 14:51:57.571301 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 14:51:57.572441 (+ 1140us) server_negotiation.cc:407] Connection header received
0927 14:51:57.572632 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:51:57.572636 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:51:57.572692 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:51:57.572750 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:51:57.573400 (+ 650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:51:57.574210 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:51:57.575466 (+ 1256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:51:57.575827 (+ 361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:51:57.575903 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:51:57.577357 (+ 1454us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:51:57.577361 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:51:57.577364 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 14:51:57.577389 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:51:57.579056 (+ 1667us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:51:57.597972 (+ 18916us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:51:57.597975 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:51:57.597977 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:51:57.598068 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:51:57.598375 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:51:57.598379 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:51:57.598380 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:51:57.598592 (+ 212us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:51:57.598668 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0927 14:51:57.598970 (+ 302us) server_negotiation.cc:299] Negotiation successful
0927 14:51:57.599085 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":127}
I20260927 14:51:57.599635 7239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:51:57.572055 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44616 (local address 127.0.98.60:38741)
0927 14:51:57.572173 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0927 14:51:57.572178 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 14:51:57.574023 (+ 1845us) server_negotiation.cc:407] Connection header received
0927 14:51:57.574177 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:51:57.574182 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:51:57.574220 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:51:57.574297 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:51:57.575138 (+ 841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:51:57.575838 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:51:57.576968 (+ 1130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:51:57.577274 (+ 306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:51:57.577395 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:51:57.578705 (+ 1310us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:51:57.578709 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:51:57.578712 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 14:51:57.578736 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:51:57.580210 (+ 1474us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:51:57.597913 (+ 17703us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:51:57.597918 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:51:57.597923 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:51:57.598001 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:51:57.598279 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:51:57.598284 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:51:57.598289 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:51:57.598505 (+ 216us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:51:57.598591 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0927 14:51:57.599354 (+ 763us) server_negotiation.cc:299] Negotiation successful
0927 14:51:57.599478 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":19,"server-negotiator.queue_time_us":61}
I20260927 14:51:57.602121 7262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:51:57.570915 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36527 (local address 127.0.0.1:40372)
0927 14:51:57.572355 (+ 1440us) negotiation.cc:107] Waiting for socket to connect
0927 14:51:57.572371 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 14:51:57.572515 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:51:57.572901 (+ 386us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:51:57.572904 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:51:57.573135 (+ 231us) client_negotiation.cc:189] Negotiated authn=SASL
0927 14:51:57.573268 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:51:57.573276 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:51:57.574349 (+ 1073us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:51:57.574353 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:51:57.575328 (+ 975us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:51:57.575339 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:51:57.576070 (+ 731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:51:57.576074 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:51:57.576170 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:51:57.576721 (+ 551us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 14:51:57.576735 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0927 14:51:57.577209 (+ 474us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 14:51:57.579190 (+ 1981us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 14:51:57.579194 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 14:51:57.579196 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 14:51:57.597854 (+ 18658us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 14:51:57.598177 (+ 323us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 14:51:57.598179 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 14:51:57.598181 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 14:51:57.598254 (+ 73us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 14:51:57.598715 (+ 461us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 14:51:57.598718 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 14:51:57.598800 (+ 82us) client_negotiation.cc:769] Sending connection context
0927 14:51:57.601775 (+ 2975us) client_negotiation.cc:240] Negotiation successful
0927 14:51:57.601919 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1370,"mutex_wait_us":398,"thread_start_us":108,"threads_started":1}
I20260927 14:51:57.604082 7241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:51:57.571305 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52194 (local address 127.0.98.61:37119)
0927 14:51:57.571431 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0927 14:51:57.571437 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 14:51:57.571786 (+ 349us) server_negotiation.cc:407] Connection header received
0927 14:51:57.571991 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:51:57.571994 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:51:57.572059 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:51:57.572137 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:51:57.572878 (+ 741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:51:57.574066 (+ 1188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:51:57.575314 (+ 1248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:51:57.575683 (+ 369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:51:57.575772 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:51:57.600449 (+ 24677us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:51:57.600454 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:51:57.600458 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 14:51:57.600490 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:51:57.602289 (+ 1799us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:51:57.602764 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:51:57.602768 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:51:57.602770 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:51:57.602866 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:51:57.603165 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:51:57.603168 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:51:57.603170 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:51:57.603375 (+ 205us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:51:57.603445 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0927 14:51:57.603786 (+ 341us) server_negotiation.cc:299] Negotiation successful
0927 14:51:57.603937 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260927 14:51:57.604660 7260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:51:57.570915 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:37119 (local address 127.0.0.1:52194)
0927 14:51:57.571696 (+ 781us) negotiation.cc:107] Waiting for socket to connect
0927 14:51:57.571712 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 14:51:57.571867 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:51:57.572299 (+ 432us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:51:57.572303 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:51:57.572543 (+ 240us) client_negotiation.cc:189] Negotiated authn=SASL
0927 14:51:57.572712 (+ 169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:51:57.572722 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:51:57.574203 (+ 1481us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:51:57.574207 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:51:57.575178 (+ 971us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:51:57.575185 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:51:57.575945 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:51:57.575950 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:51:57.576048 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:51:57.597502 (+ 21454us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 14:51:57.597521 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0927 14:51:57.600260 (+ 2739us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 14:51:57.602412 (+ 2152us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 14:51:57.602416 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 14:51:57.602420 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 14:51:57.602650 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 14:51:57.602975 (+ 325us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 14:51:57.602978 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 14:51:57.602980 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 14:51:57.603061 (+ 81us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 14:51:57.603535 (+ 474us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 14:51:57.603538 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 14:51:57.603613 (+ 75us) client_negotiation.cc:769] Sending connection context
0927 14:51:57.604317 (+ 704us) client_negotiation.cc:240] Negotiation successful
0927 14:51:57.604474 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":685,"mutex_wait_us":48,"thread_start_us":86,"threads_started":1}
Sep 27 14:51:57 dist-test-slave-xk1g krb5kdc[6473](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790520716, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
I20260927 14:51:57.613624 7239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:51:57.606474 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44632 (local address 127.0.98.60:38741)
0927 14:51:57.606568 (+ 94us) server_negotiation.cc:206] Beginning negotiation
0927 14:51:57.606572 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 14:51:57.606645 (+ 73us) server_negotiation.cc:407] Connection header received
0927 14:51:57.606801 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:51:57.606812 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:51:57.606860 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:51:57.606944 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:51:57.607318 (+ 374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:51:57.607801 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:51:57.608562 (+ 761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:51:57.608774 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:51:57.608876 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:51:57.610737 (+ 1861us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:51:57.610741 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:51:57.610743 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 14:51:57.610765 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:51:57.612093 (+ 1328us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:51:57.612507 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:51:57.612512 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:51:57.612514 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:51:57.612585 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:51:57.612864 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:51:57.612867 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:51:57.612869 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:51:57.613020 (+ 151us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:51:57.613156 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0927 14:51:57.613410 (+ 254us) server_negotiation.cc:299] Negotiation successful
0927 14:51:57.613498 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260927 14:51:57.613631 7075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:51:57.606438 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:38741 (local address 127.0.0.1:44632)
0927 14:51:57.606591 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0927 14:51:57.606604 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 14:51:57.606684 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:51:57.606959 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:51:57.606962 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:51:57.607137 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0927 14:51:57.607212 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:51:57.607217 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:51:57.607912 (+ 695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:51:57.607917 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:51:57.608443 (+ 526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:51:57.608450 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:51:57.608878 (+ 428us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:51:57.608881 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:51:57.608975 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:51:57.609437 (+ 462us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 14:51:57.609450 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0927 14:51:57.610585 (+ 1135us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 14:51:57.612206 (+ 1621us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 14:51:57.612212 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 14:51:57.612214 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 14:51:57.612410 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 14:51:57.612677 (+ 267us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 14:51:57.612681 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 14:51:57.612683 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 14:51:57.612767 (+ 84us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 14:51:57.613148 (+ 381us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 14:51:57.613152 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 14:51:57.613256 (+ 104us) client_negotiation.cc:769] Sending connection context
0927 14:51:57.613363 (+ 107us) client_negotiation.cc:240] Negotiation successful
0927 14:51:57.613494 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":91}
I20260927 14:51:57.614786 6518 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.0.98.60:38741: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "34720d7704db415f8c02e5d750103c4a"
member_type: NON_VOTER
last_known_addr {
host: "127.0.98.60"
port: 38741
}
attrs {
promote: true
}
}
dest_uuid: "d60b79f337e343bca73eadfa1bd55f6a"
cas_config_opid_index: 5
I20260927 14:51:57.615320 6518 consensus_queue.cc:237] T 00000000000000000000000000000000 P d60b79f337e343bca73eadfa1bd55f6a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "d60b79f337e343bca73eadfa1bd55f6a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36527 } } peers { permanent_uuid: "413c118097b64cd8af3711ec05bf636a" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 37119 } attrs { promote: true } } peers { permanent_uuid: "34720d7704db415f8c02e5d750103c4a" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 38741 } attrs { promote: true } }
I20260927 14:51:57.616333 7076 raft_consensus.cc:2932] T 00000000000000000000000000000000 P d60b79f337e343bca73eadfa1bd55f6a [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 34720d7704db415f8c02e5d750103c4a (127.0.98.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "d60b79f337e343bca73eadfa1bd55f6a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36527 } } peers { permanent_uuid: "413c118097b64cd8af3711ec05bf636a" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 37119 } attrs { promote: true } } peers { permanent_uuid: "34720d7704db415f8c02e5d750103c4a" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 38741 } attrs { promote: true } } }
I20260927 14:51:57.617344 7211 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 34720d7704db415f8c02e5d750103c4a [term 0 FOLLOWER]: Advancing to term 1
I20260927 14:51:57.617916 7076 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.0.98.60:38741
I20260927 14:51:57.618072 7076 sys_catalog.cc:455] T 00000000000000000000000000000000 P d60b79f337e343bca73eadfa1bd55f6a [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "d60b79f337e343bca73eadfa1bd55f6a" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "d60b79f337e343bca73eadfa1bd55f6a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36527 } } peers { permanent_uuid: "413c118097b64cd8af3711ec05bf636a" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 37119 } attrs { promote: true } } peers { permanent_uuid: "34720d7704db415f8c02e5d750103c4a" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 38741 } attrs { promote: true } } }
I20260927 14:51:57.618155 7076 sys_catalog.cc:458] T 00000000000000000000000000000000 P d60b79f337e343bca73eadfa1bd55f6a [sys.catalog]: This master's current role is: LEADER
I20260927 14:51:57.618795 7085 master.cc:537] Master@127.0.98.60:38741 shutting down...
I20260927 14:51:57.618772 7211 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 34720d7704db415f8c02e5d750103c4a [term 1 FOLLOWER]: Refusing update from remote peer d60b79f337e343bca73eadfa1bd55f6a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260927 14:51:57.619333 7077 consensus_queue.cc:1035] T 00000000000000000000000000000000 P d60b79f337e343bca73eadfa1bd55f6a [LEADER]: Connected to new peer: Peer: permanent_uuid: "34720d7704db415f8c02e5d750103c4a" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 38741 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.000s
W20260927 14:51:57.621383 6502 consensus_peers.cc:487] T 00000000000000000000000000000000 P d60b79f337e343bca73eadfa1bd55f6a -> Peer 34720d7704db415f8c02e5d750103c4a (127.0.98.60:38741): Couldn't send request to peer 34720d7704db415f8c02e5d750103c4a. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260927 14:51:57.623277 7085 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 34720d7704db415f8c02e5d750103c4a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 14:51:57.623541 7085 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 34720d7704db415f8c02e5d750103c4a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 14:51:57.623672 7085 tablet_replica.cc:330] T 00000000000000000000000000000000 P 34720d7704db415f8c02e5d750103c4a: stopping tablet replica
I20260927 14:51:57.625849 7075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:51:57.616503 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:37119 (local address 127.0.0.1:52202)
0927 14:51:57.621543 (+ 5040us) negotiation.cc:107] Waiting for socket to connect
0927 14:51:57.621556 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 14:51:57.621629 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:51:57.622000 (+ 371us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:51:57.622004 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:51:57.622024 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 14:51:57.622101 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:51:57.622106 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:51:57.623051 (+ 945us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:51:57.623057 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:51:57.624341 (+ 1284us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:51:57.624353 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:51:57.625408 (+ 1055us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:51:57.625411 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:51:57.625510 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:51:57.625553 (+ 43us) client_negotiation.cc:769] Sending connection context
0927 14:51:57.625660 (+ 107us) client_negotiation.cc:240] Negotiation successful
0927 14:51:57.625726 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4951}
I20260927 14:51:57.625851 7241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:51:57.616702 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52202 (local address 127.0.98.61:37119)
0927 14:51:57.620964 (+ 4262us) server_negotiation.cc:206] Beginning negotiation
0927 14:51:57.620969 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 14:51:57.621710 (+ 741us) server_negotiation.cc:407] Connection header received
0927 14:51:57.621760 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:51:57.621763 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:51:57.621834 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:51:57.621931 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 14:51:57.622209 (+ 278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:51:57.622956 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:51:57.624484 (+ 1528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:51:57.625093 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:51:57.625193 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:51:57.625343 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0927 14:51:57.625674 (+ 331us) server_negotiation.cc:299] Negotiation successful
0927 14:51:57.625740 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4167}
I20260927 14:51:57.626756 7129 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 413c118097b64cd8af3711ec05bf636a [term 1 LEARNER]: Refusing update from remote peer d60b79f337e343bca73eadfa1bd55f6a: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260927 14:51:57.627295 7077 consensus_queue.cc:1035] T 00000000000000000000000000000000 P d60b79f337e343bca73eadfa1bd55f6a [LEADER]: Connected to new peer: Peer: permanent_uuid: "413c118097b64cd8af3711ec05bf636a" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 37119 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260927 14:51:57.629245 7129 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 413c118097b64cd8af3711ec05bf636a [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 34720d7704db415f8c02e5d750103c4a (127.0.98.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "d60b79f337e343bca73eadfa1bd55f6a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36527 } } peers { permanent_uuid: "413c118097b64cd8af3711ec05bf636a" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 37119 } attrs { promote: true } } peers { permanent_uuid: "34720d7704db415f8c02e5d750103c4a" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 38741 } attrs { promote: true } } }
I20260927 14:51:57.630779 7265 sys_catalog.cc:455] T 00000000000000000000000000000000 P 413c118097b64cd8af3711ec05bf636a [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "d60b79f337e343bca73eadfa1bd55f6a" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "d60b79f337e343bca73eadfa1bd55f6a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36527 } } peers { permanent_uuid: "413c118097b64cd8af3711ec05bf636a" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 37119 } attrs { promote: true } } peers { permanent_uuid: "34720d7704db415f8c02e5d750103c4a" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 38741 } attrs { promote: true } } }
I20260927 14:51:57.630968 7265 sys_catalog.cc:458] T 00000000000000000000000000000000 P 413c118097b64cd8af3711ec05bf636a [sys.catalog]: This master's current role is: LEARNER
I20260927 14:51:57.631235 7158 sys_catalog.cc:455] T 00000000000000000000000000000000 P 413c118097b64cd8af3711ec05bf636a [sys.catalog]: SysCatalogTable state changed. Reason: New leader d60b79f337e343bca73eadfa1bd55f6a. Latest consensus state: current_term: 1 leader_uuid: "d60b79f337e343bca73eadfa1bd55f6a" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "d60b79f337e343bca73eadfa1bd55f6a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36527 } } peers { permanent_uuid: "413c118097b64cd8af3711ec05bf636a" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 37119 } attrs { promote: true } } peers { permanent_uuid: "34720d7704db415f8c02e5d750103c4a" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 38741 } attrs { promote: true } } }
I20260927 14:51:57.631336 7158 sys_catalog.cc:458] T 00000000000000000000000000000000 P 413c118097b64cd8af3711ec05bf636a [sys.catalog]: This master's current role is: LEARNER
I20260927 14:51:57.631516 7085 master.cc:552] Master@127.0.98.60:38741 shutdown complete.
I20260927 14:51:57.631594 7085 master_runner.cc:305] Clearing existing system tablet
I20260927 14:51:57.631877 7085 fs_manager.cc:359] Metadata directory not provided
I20260927 14:51:57.631971 7085 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 14:51:57.632925 7077 raft_consensus.cc:1050] T 00000000000000000000000000000000 P d60b79f337e343bca73eadfa1bd55f6a: attempting to promote NON_VOTER 413c118097b64cd8af3711ec05bf636a to VOTER
I20260927 14:51:57.633402 7085 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260927 14:51:57.633460 7077 consensus_queue.cc:237] T 00000000000000000000000000000000 P d60b79f337e343bca73eadfa1bd55f6a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "d60b79f337e343bca73eadfa1bd55f6a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36527 } } peers { permanent_uuid: "413c118097b64cd8af3711ec05bf636a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37119 } attrs { promote: false } } peers { permanent_uuid: "34720d7704db415f8c02e5d750103c4a" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 38741 } attrs { promote: true } }
I20260927 14:51:57.634392 7267 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 14:51:57.634629 7085 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790520684602251-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 14:51:57.634760 7085 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 14:51:57.634794 7085 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790520684602251-392-0/minicluster-data/master-2/data,/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790520684602251-392-0/minicluster-data/master-2/wal
uuid: "34720d7704db415f8c02e5d750103c4a"
format_stamp: "Formatted at 2026-09-27 14:51:57 on dist-test-slave-xk1g"
server_key: "5a3ba0183b88af4df2206b6556f1cc46"
server_key_iv: "78c99eb03462845affb95889baf526c4"
server_key_version: "encryptionkey@0"
I20260927 14:51:57.635082 7075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:51:57.634730 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:38741 (local address 127.0.0.1:44638)
0927 14:51:57.634944 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0927 14:51:57.635012 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:38741: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":145}
I20260927 14:51:57.635543 7129 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 413c118097b64cd8af3711ec05bf636a [term 1 LEARNER]: Refusing update from remote peer d60b79f337e343bca73eadfa1bd55f6a: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260927 14:51:57.635926 7076 consensus_queue.cc:1035] T 00000000000000000000000000000000 P d60b79f337e343bca73eadfa1bd55f6a [LEADER]: Connected to new peer: Peer: permanent_uuid: "413c118097b64cd8af3711ec05bf636a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37119 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260927 14:51:57.636780 7085 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 34720d7704db415f8c02e5d750103c4a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 14:51:57.636976 7075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:51:57.636710 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:38741 (local address 127.0.0.1:44640)
0927 14:51:57.636876 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0927 14:51:57.636923 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:38741: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110}
W20260927 14:51:57.637521 6502 consensus_peers.cc:487] T 00000000000000000000000000000000 P d60b79f337e343bca73eadfa1bd55f6a -> Peer 34720d7704db415f8c02e5d750103c4a (127.0.98.60:38741): Couldn't send request to peer 34720d7704db415f8c02e5d750103c4a. Status: Network error: Client connection negotiation failed: client connection to 127.0.98.60:38741: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260927 14:51:57.638417 7076 raft_consensus.cc:2932] T 00000000000000000000000000000000 P d60b79f337e343bca73eadfa1bd55f6a [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 413c118097b64cd8af3711ec05bf636a (127.0.98.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "d60b79f337e343bca73eadfa1bd55f6a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36527 } } peers { permanent_uuid: "413c118097b64cd8af3711ec05bf636a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37119 } attrs { promote: false } } peers { permanent_uuid: "34720d7704db415f8c02e5d750103c4a" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 38741 } attrs { promote: true } } }
I20260927 14:51:57.639140 7085 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 34720d7704db415f8c02e5d750103c4a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260927 14:51:57.639209 7085 log.cc:1196] T 00000000000000000000000000000000 P 34720d7704db415f8c02e5d750103c4a: Deleting WAL directory at /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790520684602251-392-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260927 14:51:57.639458 7085 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 34720d7704db415f8c02e5d750103c4a: Deleting consensus metadata
I20260927 14:51:57.639654 7085 master_runner.cc:315] Copying system tablet from 127.0.98.62:36527
I20260927 14:51:57.639844 7076 sys_catalog.cc:455] T 00000000000000000000000000000000 P d60b79f337e343bca73eadfa1bd55f6a [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "d60b79f337e343bca73eadfa1bd55f6a" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "d60b79f337e343bca73eadfa1bd55f6a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36527 } } peers { permanent_uuid: "413c118097b64cd8af3711ec05bf636a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37119 } attrs { promote: false } } peers { permanent_uuid: "34720d7704db415f8c02e5d750103c4a" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 38741 } attrs { promote: true } } }
I20260927 14:51:57.639932 7076 sys_catalog.cc:458] T 00000000000000000000000000000000 P d60b79f337e343bca73eadfa1bd55f6a [sys.catalog]: This master's current role is: LEADER
I20260927 14:51:57.640064 7076 sys_catalog.cc:455] T 00000000000000000000000000000000 P d60b79f337e343bca73eadfa1bd55f6a [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "d60b79f337e343bca73eadfa1bd55f6a" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "d60b79f337e343bca73eadfa1bd55f6a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36527 } } peers { permanent_uuid: "413c118097b64cd8af3711ec05bf636a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37119 } attrs { promote: false } } peers { permanent_uuid: "34720d7704db415f8c02e5d750103c4a" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 38741 } attrs { promote: true } } }
I20260927 14:51:57.640147 7076 sys_catalog.cc:458] T 00000000000000000000000000000000 P d60b79f337e343bca73eadfa1bd55f6a [sys.catalog]: This master's current role is: LEADER
I20260927 14:51:57.640717 7129 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 413c118097b64cd8af3711ec05bf636a [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 413c118097b64cd8af3711ec05bf636a (127.0.98.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "d60b79f337e343bca73eadfa1bd55f6a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36527 } } peers { permanent_uuid: "413c118097b64cd8af3711ec05bf636a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37119 } attrs { promote: false } } peers { permanent_uuid: "34720d7704db415f8c02e5d750103c4a" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 38741 } attrs { promote: true } } }
I20260927 14:51:57.641440 7085 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 34720d7704db415f8c02e5d750103c4a: tablet copy: Beginning tablet copy session from remote peer at address 127.0.98.62:36527
I20260927 14:51:57.642314 7158 sys_catalog.cc:455] T 00000000000000000000000000000000 P 413c118097b64cd8af3711ec05bf636a [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "d60b79f337e343bca73eadfa1bd55f6a" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "d60b79f337e343bca73eadfa1bd55f6a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36527 } } peers { permanent_uuid: "413c118097b64cd8af3711ec05bf636a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37119 } attrs { promote: false } } peers { permanent_uuid: "34720d7704db415f8c02e5d750103c4a" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 38741 } attrs { promote: true } } }
I20260927 14:51:57.642437 7158 sys_catalog.cc:458] T 00000000000000000000000000000000 P 413c118097b64cd8af3711ec05bf636a [sys.catalog]: This master's current role is: FOLLOWER
I20260927 14:51:57.650017 6557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:51:57.642132 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40376 (local address 127.0.98.62:36527)
0927 14:51:57.642229 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0927 14:51:57.642233 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 14:51:57.643121 (+ 888us) server_negotiation.cc:407] Connection header received
0927 14:51:57.643258 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:51:57.643262 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:51:57.643310 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:51:57.643404 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:51:57.643842 (+ 438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:51:57.644361 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:51:57.645338 (+ 977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:51:57.645566 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:51:57.645676 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:51:57.646775 (+ 1099us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:51:57.646778 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:51:57.646781 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 14:51:57.646804 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:51:57.648319 (+ 1515us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:51:57.648754 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:51:57.648758 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:51:57.648760 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:51:57.648840 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:51:57.649191 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:51:57.649194 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:51:57.649196 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:51:57.649382 (+ 186us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:51:57.649486 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0927 14:51:57.649768 (+ 282us) server_negotiation.cc:299] Negotiation successful
0927 14:51:57.649870 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260927 14:51:57.650054 7273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:51:57.642338 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36527 (local address 127.0.0.1:40376)
0927 14:51:57.643029 (+ 691us) negotiation.cc:107] Waiting for socket to connect
0927 14:51:57.643041 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 14:51:57.643167 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:51:57.643421 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:51:57.643424 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:51:57.643634 (+ 210us) client_negotiation.cc:189] Negotiated authn=SASL
0927 14:51:57.643722 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:51:57.643728 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:51:57.644468 (+ 740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:51:57.644475 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:51:57.645223 (+ 748us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:51:57.645232 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:51:57.645679 (+ 447us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:51:57.645682 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:51:57.645776 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:51:57.646322 (+ 546us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 14:51:57.646335 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0927 14:51:57.646634 (+ 299us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 14:51:57.648448 (+ 1814us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 14:51:57.648452 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 14:51:57.648454 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 14:51:57.648653 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 14:51:57.648940 (+ 287us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 14:51:57.648943 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 14:51:57.648945 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 14:51:57.649030 (+ 85us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 14:51:57.649476 (+ 446us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 14:51:57.649479 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 14:51:57.649599 (+ 120us) client_negotiation.cc:769] Sending connection context
0927 14:51:57.649705 (+ 106us) client_negotiation.cc:240] Negotiation successful
0927 14:51:57.649859 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":632,"thread_start_us":60,"threads_started":1}
I20260927 14:51:57.650678 6538 tablet_copy_service.cc:140] P d60b79f337e343bca73eadfa1bd55f6a: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 34720d7704db415f8c02e5d750103c4a ({username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:40376)
I20260927 14:51:57.650774 6538 tablet_copy_service.cc:161] P d60b79f337e343bca73eadfa1bd55f6a: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 34720d7704db415f8c02e5d750103c4a at {username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:40376: session id = 34720d7704db415f8c02e5d750103c4a-00000000000000000000000000000000
I20260927 14:51:57.651528 6538 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P d60b79f337e343bca73eadfa1bd55f6a: Tablet Copy: opened 0 blocks and 1 log segments
I20260927 14:51:57.652673 7085 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260927 14:51:57.654511 7085 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 34720d7704db415f8c02e5d750103c4a: tablet copy: Starting download of 0 data blocks...
I20260927 14:51:57.654711 7085 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 34720d7704db415f8c02e5d750103c4a: tablet copy: Starting download of 1 WAL segments...
I20260927 14:51:57.656070 7085 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 34720d7704db415f8c02e5d750103c4a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260927 14:51:57.657676 6538 tablet_copy_service.cc:342] P d60b79f337e343bca73eadfa1bd55f6a: Request end of tablet copy session 34720d7704db415f8c02e5d750103c4a-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:40376
I20260927 14:51:57.657748 6538 tablet_copy_service.cc:433] P d60b79f337e343bca73eadfa1bd55f6a: ending tablet copy session 34720d7704db415f8c02e5d750103c4a-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 34720d7704db415f8c02e5d750103c4a
I20260927 14:51:57.663461 7085 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 14:51:57.665076 7277 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 14:51:57.665104 7274 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 14:51:57.669274 7085 server_base.cc:1034] running on GCE node
W20260927 14:51:57.669757 7275 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 14:51:57.670035 7085 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 14:51:57.670375 7085 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 14:51:57.671514 7085 hybrid_clock.cc:648] HybridClock initialized: now 1790520717671503 us; error 30 us; skew 500 ppm
Sep 27 14:51:57 dist-test-slave-xk1g krb5kdc[6473](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790520717, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 14:51:57.672978 7085 init.cc:377] Logged in from keytab as oryx/127.0.98.60@KRBTEST.COM (short username oryx)
I20260927 14:51:57.673460 7085 webserver.cc:460] Webserver started at http://127.0.98.60:37565/ using document root <none> and password file <none>
I20260927 14:51:57.673640 7085 fs_manager.cc:359] Metadata directory not provided
I20260927 14:51:57.673696 7085 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 14:51:57.674757 7085 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260927 14:51:57.677321 7284 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 14:51:57.677619 7085 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s
I20260927 14:51:57.677681 7085 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790520684602251-392-0/minicluster-data/master-2/data,/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790520684602251-392-0/minicluster-data/master-2/wal
uuid: "34720d7704db415f8c02e5d750103c4a"
format_stamp: "Formatted at 2026-09-27 14:51:57 on dist-test-slave-xk1g"
server_key: "5a3ba0183b88af4df2206b6556f1cc46"
server_key_iv: "78c99eb03462845affb95889baf526c4"
server_key_version: "encryptionkey@0"
I20260927 14:51:57.677758 7085 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790520684602251-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790520684602251-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790520684602251-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 14:51:57.728859 7085 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 14:51:57.731073 7085 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 14:51:57.735905 7085 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:38741
I20260927 14:51:57.735921 7336 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:38741 every 8 connection(s)
I20260927 14:51:57.736527 7085 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790520684602251-392-0/minicluster-data/master-2/data/info.pb
I20260927 14:51:57.738000 7337 sys_catalog.cc:263] Verifying existing consensus state
I20260927 14:51:57.739148 7337 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 34720d7704db415f8c02e5d750103c4a: Bootstrap starting.
I20260927 14:51:57.748492 7337 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 34720d7704db415f8c02e5d750103c4a: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 14:51:57.748769 7337 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 34720d7704db415f8c02e5d750103c4a: Bootstrap complete.
I20260927 14:51:57.749398 7337 raft_consensus.cc:348] T 00000000000000000000000000000000 P 34720d7704db415f8c02e5d750103c4a [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "d60b79f337e343bca73eadfa1bd55f6a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36527 } } peers { permanent_uuid: "413c118097b64cd8af3711ec05bf636a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37119 } attrs { promote: false } } peers { permanent_uuid: "34720d7704db415f8c02e5d750103c4a" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 38741 } attrs { promote: true } }
I20260927 14:51:57.749532 7337 raft_consensus.cc:729] T 00000000000000000000000000000000 P 34720d7704db415f8c02e5d750103c4a [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 34720d7704db415f8c02e5d750103c4a, State: Initialized, Role: LEARNER
I20260927 14:51:57.749644 7337 consensus_queue.cc:260] T 00000000000000000000000000000000 P 34720d7704db415f8c02e5d750103c4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "d60b79f337e343bca73eadfa1bd55f6a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36527 } } peers { permanent_uuid: "413c118097b64cd8af3711ec05bf636a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37119 } attrs { promote: false } } peers { permanent_uuid: "34720d7704db415f8c02e5d750103c4a" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 38741 } attrs { promote: true } }
I20260927 14:51:57.750387 7343 sys_catalog.cc:455] T 00000000000000000000000000000000 P 34720d7704db415f8c02e5d750103c4a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "d60b79f337e343bca73eadfa1bd55f6a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36527 } } peers { permanent_uuid: "413c118097b64cd8af3711ec05bf636a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37119 } attrs { promote: false } } peers { permanent_uuid: "34720d7704db415f8c02e5d750103c4a" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 38741 } attrs { promote: true } } }
I20260927 14:51:57.750551 7337 sys_catalog.cc:564] T 00000000000000000000000000000000 P 34720d7704db415f8c02e5d750103c4a [sys.catalog]: configured and running, proceeding with master startup.
I20260927 14:51:57.750566 7343 sys_catalog.cc:458] T 00000000000000000000000000000000 P 34720d7704db415f8c02e5d750103c4a [sys.catalog]: This master's current role is: LEARNER
W20260927 14:51:57.750922 7263 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 14:51:57.751245 7263 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 14:51:57.751312 7263 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260927 14:51:57.754258 7354 catalog_manager.cc:1259] Loaded cluster ID: 3ea1e4dfc1684ee8ba96f40d83c0eb15
I20260927 14:51:57.754335 7354 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 34720d7704db415f8c02e5d750103c4a: loading cluster ID for follower catalog manager: success
I20260927 14:51:57.755868 7354 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 34720d7704db415f8c02e5d750103c4a: acquiring CA information for follower catalog manager: success
I20260927 14:51:57.757164 7354 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 34720d7704db415f8c02e5d750103c4a: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260927 14:51:57.757427 7263 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 14:51:57.757500 7263 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 14:51:57.757627 7263 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 14:51:57.765460 7263 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44229
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790520684602251-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790520684602251-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790520684602251-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:43017
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790520684602251-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=33431
--tserver_master_addrs=127.0.98.62:36527,127.0.98.61:37119,127.0.98.60:38741
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7263
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790520684602251-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:15:17 UTC on ccycloud.quasar-ledxfe.root.comops.site
build id 4676134
I20260927 14:51:57.766608 7263 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 14:51:57.767781 7263 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 14:51:57.779441 7356 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 14:51:57.779587 7357 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 14:51:57.779667 7263 server_base.cc:1034] running on GCE node
W20260927 14:51:57.779906 7359 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 14:51:57.780284 7263 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 14:51:57.780849 7263 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 14:51:57.781999 7263 hybrid_clock.cc:648] HybridClock initialized: now 1790520717781977 us; error 36 us; skew 500 ppm
Sep 27 14:51:57 dist-test-slave-xk1g krb5kdc[6473](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790520717, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 14:51:57.784590 7263 init.cc:377] Logged in from keytab as oryx/127.0.98.1@KRBTEST.COM (short username oryx)
I20260927 14:51:57.785526 7263 webserver.cc:460] Webserver started at http://127.0.98.1:33431/ using document root <none> and password file <none>
I20260927 14:51:57.786058 7263 fs_manager.cc:359] Metadata directory not provided
I20260927 14:51:57.786150 7263 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 14:51:57.789737 7263 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260927 14:51:57.791656 7366 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 14:51:57.792997 7263 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 14:51:57.793123 7263 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790520684602251-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790520684602251-392-0/minicluster-data/ts-0/wal
uuid: "f37a65090f8b4b8f9a4facdb930c9eec"
format_stamp: "Formatted at 2026-09-27 14:51:56 on dist-test-slave-xk1g"
server_key: "814b5ad532276f62ea9f3dc83580cd70"
server_key_iv: "d3a228153c70238fb4e9d69303bde9c5"
server_key_version: "encryptionkey@0"
I20260927 14:51:57.793757 7263 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790520684602251-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790520684602251-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790520684602251-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 14:51:57.794390 6843 heartbeater.cc:503] Master 127.0.98.62:36527 was elected leader, sending a full tablet report...
I20260927 14:51:57.809448 7263 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 14:51:57.812448 7263 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 14:51:57.812650 7263 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 14:51:57.813292 7263 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 14:51:57.814141 7263 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 14:51:57.814215 7263 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 14:51:57.814280 7263 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 14:51:57.814332 7263 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 14:51:57.822832 7263 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:43017
I20260927 14:51:57.822912 7479 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:43017 every 8 connection(s)
I20260927 14:51:57.823824 7263 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790520684602251-392-0/minicluster-data/ts-0/data/info.pb
I20260927 14:51:57.827566 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6936m5/build/debug/bin/kudu as pid 7263
I20260927 14:51:57.827751 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6936m5/build/debug/bin/kudu with pid 6714
I20260927 14:51:57.835589 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6936m5/build/debug/bin/kudu
/tmp/dist-test-task6936m5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790520684602251-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790520684602251-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790520684602251-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790520684602251-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:34621
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:36527
--webserver_port=36055
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790520684602251-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44229
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.98.62:36527,127.0.98.61:37119
--tserver_master_addrs=127.0.98.62:36527,127.0.98.61:37119,127.0.98.60:38741 with env {KRB5CCNAME=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790520684602251-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790520684602251-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790520684602251-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 14:51:57 dist-test-slave-xk1g krb5kdc[6473](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790520717, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
Sep 27 14:51:57 dist-test-slave-xk1g krb5kdc[6473](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790520717, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
Sep 27 14:51:57 dist-test-slave-xk1g krb5kdc[6473](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790520717, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260927 14:51:57.844192 7241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:51:57.825592 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:57247 (local address 127.0.98.61:37119)
0927 14:51:57.825726 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0927 14:51:57.825730 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 14:51:57.827222 (+ 1492us) server_negotiation.cc:407] Connection header received
0927 14:51:57.829956 (+ 2734us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:51:57.829959 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:51:57.830010 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:51:57.830067 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:51:57.832011 (+ 1944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:51:57.832810 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:51:57.834820 (+ 2010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:51:57.835200 (+ 380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:51:57.835302 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:51:57.839203 (+ 3901us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:51:57.839207 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:51:57.839210 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 14:51:57.839238 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:51:57.841666 (+ 2428us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:51:57.842285 (+ 619us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:51:57.842291 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:51:57.842296 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:51:57.842375 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:51:57.842663 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:51:57.842667 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:51:57.842668 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:51:57.842945 (+ 277us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:51:57.843036 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0927 14:51:57.843909 (+ 873us) server_negotiation.cc:299] Negotiation successful
0927 14:51:57.844032 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260927 14:51:57.844197 7484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:51:57.825828 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:34383 (local address 127.0.98.60:38741)
0927 14:51:57.826147 (+ 319us) server_negotiation.cc:206] Beginning negotiation
0927 14:51:57.826177 (+ 30us) server_negotiation.cc:399] Waiting for connection header
0927 14:51:57.827214 (+ 1037us) server_negotiation.cc:407] Connection header received
0927 14:51:57.829950 (+ 2736us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:51:57.829954 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:51:57.830010 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:51:57.830070 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:51:57.832010 (+ 1940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:51:57.833097 (+ 1087us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:51:57.834833 (+ 1736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:51:57.835195 (+ 362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:51:57.835276 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:51:57.839657 (+ 4381us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:51:57.839661 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:51:57.839665 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 14:51:57.839694 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:51:57.841469 (+ 1775us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:51:57.842171 (+ 702us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:51:57.842175 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:51:57.842177 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:51:57.842266 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:51:57.842663 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:51:57.842667 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:51:57.842668 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:51:57.843193 (+ 525us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:51:57.843296 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0927 14:51:57.843909 (+ 613us) server_negotiation.cc:299] Negotiation successful
0927 14:51:57.844033 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":243,"thread_start_us":134,"threads_started":1}
I20260927 14:51:57.844282 6557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:51:57.825642 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:40683 (local address 127.0.98.62:36527)
0927 14:51:57.825763 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0927 14:51:57.825767 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 14:51:57.827214 (+ 1447us) server_negotiation.cc:407] Connection header received
0927 14:51:57.829943 (+ 2729us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:51:57.829949 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:51:57.830012 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:51:57.830068 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:51:57.832004 (+ 1936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:51:57.832922 (+ 918us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:51:57.834783 (+ 1861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:51:57.836901 (+ 2118us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:51:57.836995 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:51:57.839758 (+ 2763us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:51:57.839763 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:51:57.839769 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 14:51:57.839803 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:51:57.841649 (+ 1846us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:51:57.842369 (+ 720us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:51:57.842374 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:51:57.842379 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:51:57.842471 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:51:57.842813 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:51:57.842818 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:51:57.842821 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:51:57.843120 (+ 299us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:51:57.843218 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0927 14:51:57.843928 (+ 710us) server_negotiation.cc:299] Negotiation successful
0927 14:51:57.844063 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260927 14:51:57.845788 7485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:51:57.826013 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:38741 (local address 127.0.98.1:34383)
0927 14:51:57.827091 (+ 1078us) negotiation.cc:107] Waiting for socket to connect
0927 14:51:57.827130 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0927 14:51:57.829715 (+ 2585us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:51:57.830453 (+ 738us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:51:57.830455 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:51:57.831126 (+ 671us) client_negotiation.cc:189] Negotiated authn=SASL
0927 14:51:57.831845 (+ 719us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:51:57.831859 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:51:57.833222 (+ 1363us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:51:57.833225 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:51:57.834685 (+ 1460us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:51:57.834696 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:51:57.835515 (+ 819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:51:57.835519 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:51:57.835744 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:51:57.836425 (+ 681us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 14:51:57.836454 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0927 14:51:57.839503 (+ 3049us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 14:51:57.841591 (+ 2088us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 14:51:57.841598 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 14:51:57.841614 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0927 14:51:57.842026 (+ 412us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 14:51:57.842375 (+ 349us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 14:51:57.842378 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 14:51:57.842380 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 14:51:57.842557 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 14:51:57.843301 (+ 744us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 14:51:57.843305 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 14:51:57.843617 (+ 312us) client_negotiation.cc:769] Sending connection context
0927 14:51:57.844334 (+ 717us) client_negotiation.cc:240] Negotiation successful
0927 14:51:57.844731 (+ 397us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":936,"mutex_wait_us":436,"spinlock_wait_cycles":176256,"thread_start_us":689,"threads_started":1}
I20260927 14:51:57.845793 7487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:51:57.826013 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36527 (local address 127.0.98.1:40683)
0927 14:51:57.827129 (+ 1116us) negotiation.cc:107] Waiting for socket to connect
0927 14:51:57.827140 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 14:51:57.829705 (+ 2565us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:51:57.830446 (+ 741us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:51:57.830457 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:51:57.831088 (+ 631us) client_negotiation.cc:189] Negotiated authn=SASL
0927 14:51:57.831838 (+ 750us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:51:57.831858 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:51:57.833092 (+ 1234us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:51:57.833098 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:51:57.834659 (+ 1561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:51:57.834671 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:51:57.837208 (+ 2537us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:51:57.837213 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:51:57.837338 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:51:57.837835 (+ 497us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 14:51:57.837851 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 14:51:57.839593 (+ 1742us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 14:51:57.841775 (+ 2182us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 14:51:57.841798 (+ 23us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 14:51:57.841801 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 14:51:57.842232 (+ 431us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 14:51:57.842596 (+ 364us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 14:51:57.842601 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 14:51:57.842605 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 14:51:57.842700 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 14:51:57.843292 (+ 592us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 14:51:57.843300 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 14:51:57.843657 (+ 357us) client_negotiation.cc:769] Sending connection context
0927 14:51:57.844489 (+ 832us) client_negotiation.cc:240] Negotiation successful
0927 14:51:57.844827 (+ 338us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1034,"mutex_wait_us":500,"spinlock_wait_cycles":1249920,"thread_start_us":694,"threads_started":1}
I20260927 14:51:57.846738 7486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:51:57.826013 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:37119 (local address 127.0.98.1:57247)
0927 14:51:57.827090 (+ 1077us) negotiation.cc:107] Waiting for socket to connect
0927 14:51:57.827128 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0927 14:51:57.829715 (+ 2587us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:51:57.830446 (+ 731us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:51:57.830455 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:51:57.831134 (+ 679us) client_negotiation.cc:189] Negotiated authn=SASL
0927 14:51:57.831846 (+ 712us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:51:57.831859 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:51:57.832981 (+ 1122us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:51:57.832985 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:51:57.834687 (+ 1702us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:51:57.834696 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:51:57.835516 (+ 820us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:51:57.835519 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:51:57.836157 (+ 638us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:51:57.836559 (+ 402us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 14:51:57.836571 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 14:51:57.838957 (+ 2386us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 14:51:57.841775 (+ 2818us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 14:51:57.841795 (+ 20us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 14:51:57.841797 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 14:51:57.842182 (+ 385us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 14:51:57.842476 (+ 294us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 14:51:57.842479 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 14:51:57.842481 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 14:51:57.842557 (+ 76us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 14:51:57.843349 (+ 792us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 14:51:57.843351 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 14:51:57.843617 (+ 266us) client_negotiation.cc:769] Sending connection context
0927 14:51:57.844890 (+ 1273us) client_negotiation.cc:240] Negotiation successful
0927 14:51:57.845018 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":887,"mutex_wait_us":423,"spinlock_wait_cycles":39168,"thread_start_us":136,"threads_started":1}
I20260927 14:51:57.847272 7480 heartbeater.cc:347] Connected to a master server at 127.0.98.60:38741
I20260927 14:51:57.847286 7481 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36527
I20260927 14:51:57.847522 7480 heartbeater.cc:464] Registering TS with master...
I20260927 14:51:57.847522 7481 heartbeater.cc:464] Registering TS with master...
I20260927 14:51:57.847525 7482 heartbeater.cc:347] Connected to a master server at 127.0.98.61:37119
I20260927 14:51:57.848013 7482 heartbeater.cc:464] Registering TS with master...
I20260927 14:51:57.848091 7480 heartbeater.cc:511] Master 127.0.98.60:38741 requested a full tablet report, sending...
I20260927 14:51:57.848143 7482 heartbeater.cc:511] Master 127.0.98.61:37119 requested a full tablet report, sending...
I20260927 14:51:57.848090 7481 heartbeater.cc:511] Master 127.0.98.62:36527 requested a full tablet report, sending...
I20260927 14:51:57.849352 6518 ts_manager.cc:194] Re-registered known tserver with Master: f37a65090f8b4b8f9a4facdb930c9eec (127.0.98.1:43017)
I20260927 14:51:57.849614 7119 ts_manager.cc:194] Registered new tserver with Master: f37a65090f8b4b8f9a4facdb930c9eec (127.0.98.1:43017)
I20260927 14:51:57.849798 7301 ts_manager.cc:194] Registered new tserver with Master: f37a65090f8b4b8f9a4facdb930c9eec (127.0.98.1:43017)
I20260927 14:51:57.850054 6518 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:40683
W20260927 14:51:57.952716 7488 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 14:51:57.952953 7488 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 14:51:57.952993 7488 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 14:51:57.956948 7488 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 14:51:57.957002 7488 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 14:51:57.957105 7488 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 14:51:57.962378 7488 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44229
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790520684602251-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790520684602251-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790520684602251-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:34621
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790520684602251-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=36055
--tserver_master_addrs=127.0.98.62:36527,127.0.98.61:37119,127.0.98.60:38741
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7488
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790520684602251-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:15:17 UTC on ccycloud.quasar-ledxfe.root.comops.site
build id 4676134
I20260927 14:51:57.963248 7488 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 14:51:57.964252 7488 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 14:51:57.973299 7494 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 14:51:57.973332 7493 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 14:51:57.973306 7496 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 14:51:57.973656 7488 server_base.cc:1034] running on GCE node
I20260927 14:51:57.974099 7488 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 14:51:57.974695 7488 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 14:51:57.975862 7488 hybrid_clock.cc:648] HybridClock initialized: now 1790520717975844 us; error 26 us; skew 500 ppm
Sep 27 14:51:57 dist-test-slave-xk1g krb5kdc[6473](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790520717, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 14:51:57.978470 7488 init.cc:377] Logged in from keytab as oryx/127.0.98.2@KRBTEST.COM (short username oryx)
I20260927 14:51:57.979398 7488 webserver.cc:460] Webserver started at http://127.0.98.2:36055/ using document root <none> and password file <none>
I20260927 14:51:57.979933 7488 fs_manager.cc:359] Metadata directory not provided
I20260927 14:51:57.980041 7488 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 14:51:57.984225 7488 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260927 14:51:57.986318 7503 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 14:51:57.987491 7488 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 14:51:57.987628 7488 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790520684602251-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790520684602251-392-0/minicluster-data/ts-1/wal
uuid: "a473db7856d24a8ba3efb9bbc9e12de9"
format_stamp: "Formatted at 2026-09-27 14:51:56 on dist-test-slave-xk1g"
server_key: "2d2692160a00ee3fd176e4475d2e6a41"
server_key_iv: "db5eaf6ce4e6f481305152d0e6b40338"
server_key_version: "encryptionkey@0"
I20260927 14:51:57.988124 7488 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790520684602251-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790520684602251-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790520684602251-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 14:51:58.002493 6977 heartbeater.cc:503] Master 127.0.98.62:36527 was elected leader, sending a full tablet report...
I20260927 14:51:58.006323 7488 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 14:51:58.009274 7488 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 14:51:58.009467 7488 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 14:51:58.010105 7488 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 14:51:58.010964 7488 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 14:51:58.011036 7488 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 14:51:58.011107 7488 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 14:51:58.011159 7488 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 14:51:58.020073 7488 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:34621
I20260927 14:51:58.020112 7616 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:34621 every 8 connection(s)
I20260927 14:51:58.021039 7488 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790520684602251-392-0/minicluster-data/ts-1/data/info.pb
I20260927 14:51:58.022590 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6936m5/build/debug/bin/kudu as pid 7488
I20260927 14:51:58.022770 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6936m5/build/debug/bin/kudu with pid 6848
I20260927 14:51:58.030287 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6936m5/build/debug/bin/kudu
/tmp/dist-test-task6936m5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790520684602251-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790520684602251-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790520684602251-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790520684602251-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:38555
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:36527
--webserver_port=37503
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790520684602251-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44229
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.98.62:36527,127.0.98.61:37119
--tserver_master_addrs=127.0.98.62:36527,127.0.98.61:37119,127.0.98.60:38741 with env {KRB5CCNAME=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790520684602251-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790520684602251-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790520684602251-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 14:51:58 dist-test-slave-xk1g krb5kdc[6473](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790520717, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
Sep 27 14:51:58 dist-test-slave-xk1g krb5kdc[6473](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790520717, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
Sep 27 14:51:58 dist-test-slave-xk1g krb5kdc[6473](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790520717, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
I20260927 14:51:58.041288 6557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:51:58.024508 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:40817 (local address 127.0.98.62:36527)
0927 14:51:58.024622 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0927 14:51:58.024626 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 14:51:58.024773 (+ 147us) server_negotiation.cc:407] Connection header received
0927 14:51:58.026258 (+ 1485us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:51:58.026265 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:51:58.026329 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:51:58.026386 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:51:58.028409 (+ 2023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:51:58.029307 (+ 898us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:51:58.031250 (+ 1943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:51:58.031584 (+ 334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:51:58.031666 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:51:58.037171 (+ 5505us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:51:58.037177 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:51:58.037182 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 14:51:58.037216 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:51:58.039178 (+ 1962us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:51:58.039693 (+ 515us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:51:58.039696 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:51:58.039698 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:51:58.039772 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:51:58.040086 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:51:58.040089 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:51:58.040091 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:51:58.040310 (+ 219us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:51:58.040374 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0927 14:51:58.040976 (+ 602us) server_negotiation.cc:299] Negotiation successful
0927 14:51:58.041123 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260927 14:51:58.041288 7241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:51:58.022950 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:48511 (local address 127.0.98.61:37119)
0927 14:51:58.023690 (+ 740us) server_negotiation.cc:206] Beginning negotiation
0927 14:51:58.023697 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 14:51:58.025572 (+ 1875us) server_negotiation.cc:407] Connection header received
0927 14:51:58.026245 (+ 673us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:51:58.026249 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:51:58.026301 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:51:58.026354 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:51:58.028403 (+ 2049us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:51:58.029234 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:51:58.031230 (+ 1996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:51:58.031625 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:51:58.031715 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:51:58.036446 (+ 4731us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:51:58.036449 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:51:58.036451 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 14:51:58.036475 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:51:58.038797 (+ 2322us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:51:58.039424 (+ 627us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:51:58.039430 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:51:58.039434 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:51:58.039521 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:51:58.039918 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:51:58.039921 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:51:58.039923 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:51:58.040160 (+ 237us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:51:58.040234 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0927 14:51:58.040976 (+ 742us) server_negotiation.cc:299] Negotiation successful
0927 14:51:58.041123 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":625}
I20260927 14:51:58.041560 7484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:51:58.023580 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:53109 (local address 127.0.98.60:38741)
0927 14:51:58.024055 (+ 475us) server_negotiation.cc:206] Beginning negotiation
0927 14:51:58.024062 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 14:51:58.025485 (+ 1423us) server_negotiation.cc:407] Connection header received
0927 14:51:58.026246 (+ 761us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:51:58.026249 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:51:58.026301 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:51:58.026354 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:51:58.028410 (+ 2056us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:51:58.029270 (+ 860us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:51:58.031249 (+ 1979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:51:58.031611 (+ 362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:51:58.031679 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:51:58.037330 (+ 5651us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:51:58.037336 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:51:58.037339 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 14:51:58.037377 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:51:58.039246 (+ 1869us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:51:58.039836 (+ 590us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:51:58.039842 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:51:58.039846 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:51:58.039950 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:51:58.040300 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:51:58.040306 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:51:58.040309 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:51:58.040553 (+ 244us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:51:58.040636 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0927 14:51:58.041184 (+ 548us) server_negotiation.cc:299] Negotiation successful
0927 14:51:58.041339 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":384}
I20260927 14:51:58.042937 7621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:51:58.023327 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:38741 (local address 127.0.98.2:53109)
0927 14:51:58.025393 (+ 2066us) negotiation.cc:107] Waiting for socket to connect
0927 14:51:58.025408 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 14:51:58.026015 (+ 607us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:51:58.026768 (+ 753us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:51:58.026778 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:51:58.027435 (+ 657us) client_negotiation.cc:189] Negotiated authn=SASL
0927 14:51:58.028239 (+ 804us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:51:58.028250 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:51:58.029984 (+ 1734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:51:58.029987 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:51:58.031109 (+ 1122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:51:58.031120 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:51:58.031872 (+ 752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:51:58.031876 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:51:58.032145 (+ 269us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:51:58.033154 (+ 1009us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 14:51:58.033170 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 14:51:58.037159 (+ 3989us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 14:51:58.039385 (+ 2226us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 14:51:58.039388 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 14:51:58.039390 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 14:51:58.039711 (+ 321us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 14:51:58.040079 (+ 368us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 14:51:58.040085 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 14:51:58.040089 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 14:51:58.040187 (+ 98us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 14:51:58.040717 (+ 530us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 14:51:58.040723 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 14:51:58.040842 (+ 119us) client_negotiation.cc:769] Sending connection context
0927 14:51:58.041769 (+ 927us) client_negotiation.cc:240] Negotiation successful
0927 14:51:58.041951 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1945,"mutex_wait_us":499,"spinlock_wait_cycles":39936,"thread_start_us":115,"threads_started":1}
I20260927 14:51:58.043181 7623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:51:58.023319 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36527 (local address 127.0.98.2:40817)
0927 14:51:58.024571 (+ 1252us) negotiation.cc:107] Waiting for socket to connect
0927 14:51:58.024622 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0927 14:51:58.026004 (+ 1382us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:51:58.026768 (+ 764us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:51:58.026780 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:51:58.027424 (+ 644us) client_negotiation.cc:189] Negotiated authn=SASL
0927 14:51:58.028236 (+ 812us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:51:58.028252 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:51:58.029459 (+ 1207us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:51:58.029466 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:51:58.031118 (+ 1652us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:51:58.031124 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:51:58.031872 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:51:58.031876 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:51:58.032144 (+ 268us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:51:58.033050 (+ 906us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 14:51:58.033134 (+ 84us) client_negotiation.cc:656] calling sasl_client_start()
0927 14:51:58.036721 (+ 3587us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 14:51:58.039299 (+ 2578us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 14:51:58.039302 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 14:51:58.039304 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 14:51:58.039585 (+ 281us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 14:51:58.039900 (+ 315us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 14:51:58.039904 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 14:51:58.039906 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 14:51:58.039984 (+ 78us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 14:51:58.040437 (+ 453us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 14:51:58.040440 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 14:51:58.040675 (+ 235us) client_negotiation.cc:769] Sending connection context
0927 14:51:58.041448 (+ 773us) client_negotiation.cc:240] Negotiation successful
0927 14:51:58.041835 (+ 387us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1051,"mutex_wait_us":649,"spinlock_wait_cycles":19072,"thread_start_us":155,"threads_started":1}
I20260927 14:51:58.042934 7622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:51:58.023319 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:37119 (local address 127.0.98.2:48511)
0927 14:51:58.025492 (+ 2173us) negotiation.cc:107] Waiting for socket to connect
0927 14:51:58.025506 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 14:51:58.026015 (+ 509us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:51:58.026768 (+ 753us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:51:58.026778 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:51:58.027447 (+ 669us) client_negotiation.cc:189] Negotiated authn=SASL
0927 14:51:58.028229 (+ 782us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:51:58.028250 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:51:58.029413 (+ 1163us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:51:58.029416 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:51:58.031087 (+ 1671us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:51:58.031101 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:51:58.031919 (+ 818us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:51:58.031926 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:51:58.032143 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:51:58.032828 (+ 685us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 14:51:58.032851 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0927 14:51:58.036271 (+ 3420us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 14:51:58.038935 (+ 2664us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 14:51:58.038943 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 14:51:58.038960 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0927 14:51:58.039303 (+ 343us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 14:51:58.039636 (+ 333us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 14:51:58.039641 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 14:51:58.039644 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 14:51:58.039812 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 14:51:58.040330 (+ 518us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 14:51:58.040340 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 14:51:58.040675 (+ 335us) client_negotiation.cc:769] Sending connection context
0927 14:51:58.042187 (+ 1512us) client_negotiation.cc:240] Negotiation successful
0927 14:51:58.042327 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2067,"mutex_wait_us":1105,"spinlock_wait_cycles":8576,"thread_start_us":104,"threads_started":1}
I20260927 14:51:58.044186 7618 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36527
I20260927 14:51:58.044430 7618 heartbeater.cc:464] Registering TS with master...
I20260927 14:51:58.044821 7619 heartbeater.cc:347] Connected to a master server at 127.0.98.61:37119
I20260927 14:51:58.044914 7619 heartbeater.cc:464] Registering TS with master...
I20260927 14:51:58.044979 7618 heartbeater.cc:511] Master 127.0.98.62:36527 requested a full tablet report, sending...
I20260927 14:51:58.045102 7619 heartbeater.cc:511] Master 127.0.98.61:37119 requested a full tablet report, sending...
I20260927 14:51:58.045544 7617 heartbeater.cc:347] Connected to a master server at 127.0.98.60:38741
I20260927 14:51:58.045629 7617 heartbeater.cc:464] Registering TS with master...
I20260927 14:51:58.045786 7617 heartbeater.cc:511] Master 127.0.98.60:38741 requested a full tablet report, sending...
I20260927 14:51:58.046164 7119 ts_manager.cc:194] Registered new tserver with Master: a473db7856d24a8ba3efb9bbc9e12de9 (127.0.98.2:34621)
I20260927 14:51:58.046420 7301 ts_manager.cc:194] Registered new tserver with Master: a473db7856d24a8ba3efb9bbc9e12de9 (127.0.98.2:34621)
I20260927 14:51:58.046651 6518 ts_manager.cc:194] Re-registered known tserver with Master: a473db7856d24a8ba3efb9bbc9e12de9 (127.0.98.2:34621)
I20260927 14:51:58.047183 6518 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:40817
W20260927 14:51:58.145339 7624 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 14:51:58.145596 7624 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 14:51:58.145638 7624 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 14:51:58.149564 7624 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 14:51:58.149616 7624 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 14:51:58.149672 7624 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 14:51:58.154985 7624 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44229
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790520684602251-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790520684602251-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790520684602251-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:38555
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790520684602251-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=37503
--tserver_master_addrs=127.0.98.62:36527,127.0.98.61:37119,127.0.98.60:38741
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7624
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790520684602251-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:15:17 UTC on ccycloud.quasar-ledxfe.root.comops.site
build id 4676134
I20260927 14:51:58.155994 7624 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 14:51:58.157012 7624 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 14:51:58.166649 7632 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 14:51:58.166657 7630 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 14:51:58.166657 7629 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 14:51:58.166996 7624 server_base.cc:1034] running on GCE node
I20260927 14:51:58.167392 7624 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 14:51:58.168032 7624 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 14:51:58.169152 7624 hybrid_clock.cc:648] HybridClock initialized: now 1790520718169118 us; error 53 us; skew 500 ppm
Sep 27 14:51:58 dist-test-slave-xk1g krb5kdc[6473](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790520718, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 14:51:58.172014 7624 init.cc:377] Logged in from keytab as oryx/127.0.98.3@KRBTEST.COM (short username oryx)
I20260927 14:51:58.172974 7624 webserver.cc:460] Webserver started at http://127.0.98.3:37503/ using document root <none> and password file <none>
I20260927 14:51:58.173576 7624 fs_manager.cc:359] Metadata directory not provided
I20260927 14:51:58.173651 7624 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 14:51:58.177532 7624 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 14:51:58.179585 7639 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 14:51:58.180814 7624 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 14:51:58.180931 7624 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790520684602251-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790520684602251-392-0/minicluster-data/ts-2/wal
uuid: "df8799b0da0c4b4e895000a21761211e"
format_stamp: "Formatted at 2026-09-27 14:51:56 on dist-test-slave-xk1g"
server_key: "9aa5d34d8927738d9765447845d19769"
server_key_iv: "32cc3ec66166373ee283f488018464c0"
server_key_version: "encryptionkey@0"
I20260927 14:51:58.181519 7624 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790520684602251-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790520684602251-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790520684602251-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 14:51:58.193941 7644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:51:58.189983 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:38741 (local address 127.0.0.1:44642)
0927 14:51:58.190209 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0927 14:51:58.190222 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 14:51:58.190330 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:51:58.190603 (+ 273us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:51:58.190605 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:51:58.190620 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 14:51:58.190709 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:51:58.190714 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:51:58.191695 (+ 981us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:51:58.191697 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:51:58.192682 (+ 985us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:51:58.192692 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:51:58.193620 (+ 928us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:51:58.193623 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:51:58.193691 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:51:58.193712 (+ 21us) client_negotiation.cc:769] Sending connection context
0927 14:51:58.193764 (+ 52us) client_negotiation.cc:240] Negotiation successful
0927 14:51:58.193822 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":65,"threads_started":1}
I20260927 14:51:58.193967 7484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:51:58.190090 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44642 (local address 127.0.98.60:38741)
0927 14:51:58.190237 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0927 14:51:58.190241 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 14:51:58.190300 (+ 59us) server_negotiation.cc:407] Connection header received
0927 14:51:58.190427 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:51:58.190430 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:51:58.190490 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:51:58.190584 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 14:51:58.190802 (+ 218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:51:58.191596 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:51:58.192794 (+ 1198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:51:58.193501 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:51:58.193600 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:51:58.193750 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0927 14:51:58.193783 (+ 33us) server_negotiation.cc:299] Negotiation successful
0927 14:51:58.193840 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260927 14:51:58.194748 7306 raft_consensus.cc:1203] T 00000000000000000000000000000000 P 34720d7704db415f8c02e5d750103c4a [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20260927 14:51:58.195178 7343 sys_catalog.cc:455] T 00000000000000000000000000000000 P 34720d7704db415f8c02e5d750103c4a [sys.catalog]: SysCatalogTable state changed. Reason: New leader d60b79f337e343bca73eadfa1bd55f6a. Latest consensus state: current_term: 1 leader_uuid: "d60b79f337e343bca73eadfa1bd55f6a" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "d60b79f337e343bca73eadfa1bd55f6a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36527 } } peers { permanent_uuid: "413c118097b64cd8af3711ec05bf636a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37119 } attrs { promote: false } } peers { permanent_uuid: "34720d7704db415f8c02e5d750103c4a" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 38741 } attrs { promote: true } } }
I20260927 14:51:58.195307 7343 sys_catalog.cc:458] T 00000000000000000000000000000000 P 34720d7704db415f8c02e5d750103c4a [sys.catalog]: This master's current role is: LEARNER
I20260927 14:51:58.199486 7624 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 14:51:58.202386 7624 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 14:51:58.202584 7624 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 14:51:58.203171 7624 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 14:51:58.204034 7624 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 14:51:58.204085 7624 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 14:51:58.204155 7624 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 14:51:58.204185 7624 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 14:51:58.212407 7624 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:38555
I20260927 14:51:58.212474 7753 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:38555 every 8 connection(s)
I20260927 14:51:58.213455 7624 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790520684602251-392-0/minicluster-data/ts-2/data/info.pb
I20260927 14:51:58.217876 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6936m5/build/debug/bin/kudu as pid 7624
Sep 27 14:51:58 dist-test-slave-xk1g krb5kdc[6473](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790520718, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
Sep 27 14:51:58 dist-test-slave-xk1g krb5kdc[6473](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790520718, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
Sep 27 14:51:58 dist-test-slave-xk1g krb5kdc[6473](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790520718, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
I20260927 14:51:58.231364 6557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:51:58.215429 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:37479 (local address 127.0.98.62:36527)
0927 14:51:58.215682 (+ 253us) server_negotiation.cc:206] Beginning negotiation
0927 14:51:58.215686 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 14:51:58.216363 (+ 677us) server_negotiation.cc:407] Connection header received
0927 14:51:58.217494 (+ 1131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:51:58.217498 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:51:58.217546 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:51:58.217599 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:51:58.219967 (+ 2368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:51:58.220716 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:51:58.222579 (+ 1863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:51:58.222917 (+ 338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:51:58.223013 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:51:58.226945 (+ 3932us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:51:58.226950 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:51:58.226952 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 14:51:58.226979 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:51:58.229021 (+ 2042us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:51:58.230109 (+ 1088us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:51:58.230114 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:51:58.230115 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:51:58.230187 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:51:58.230497 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:51:58.230500 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:51:58.230502 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:51:58.230691 (+ 189us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:51:58.230771 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0927 14:51:58.231077 (+ 306us) server_negotiation.cc:299] Negotiation successful
0927 14:51:58.231196 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":182}
I20260927 14:51:58.231364 7241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:51:58.215817 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:33799 (local address 127.0.98.61:37119)
0927 14:51:58.215951 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0927 14:51:58.215956 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 14:51:58.216865 (+ 909us) server_negotiation.cc:407] Connection header received
0927 14:51:58.217475 (+ 610us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:51:58.217479 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:51:58.217536 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:51:58.217612 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:51:58.221188 (+ 3576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:51:58.221790 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:51:58.223026 (+ 1236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:51:58.223363 (+ 337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:51:58.223437 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:51:58.226753 (+ 3316us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:51:58.226757 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:51:58.226760 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 14:51:58.226786 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:51:58.228750 (+ 1964us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:51:58.229463 (+ 713us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:51:58.229469 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:51:58.229473 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:51:58.229560 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:51:58.230058 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:51:58.230064 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:51:58.230067 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:51:58.230291 (+ 224us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:51:58.230369 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0927 14:51:58.231082 (+ 713us) server_negotiation.cc:299] Negotiation successful
0927 14:51:58.231196 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260927 14:51:58.233085 7758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:51:58.215703 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36527 (local address 127.0.98.3:37479)
0927 14:51:58.216215 (+ 512us) negotiation.cc:107] Waiting for socket to connect
0927 14:51:58.216266 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0927 14:51:58.217264 (+ 998us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:51:58.218271 (+ 1007us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:51:58.218280 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:51:58.218865 (+ 585us) client_negotiation.cc:189] Negotiated authn=SASL
0927 14:51:58.219846 (+ 981us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:51:58.219854 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:51:58.220865 (+ 1011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:51:58.220868 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:51:58.222340 (+ 1472us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:51:58.222348 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:51:58.223170 (+ 822us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:51:58.223174 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:51:58.223598 (+ 424us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:51:58.224452 (+ 854us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 14:51:58.224469 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 14:51:58.226813 (+ 2344us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 14:51:58.229769 (+ 2956us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 14:51:58.229771 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 14:51:58.229773 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 14:51:58.230004 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 14:51:58.230291 (+ 287us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 14:51:58.230295 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 14:51:58.230297 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 14:51:58.230374 (+ 77us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 14:51:58.230823 (+ 449us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 14:51:58.230825 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 14:51:58.230896 (+ 71us) client_negotiation.cc:769] Sending connection context
0927 14:51:58.231529 (+ 633us) client_negotiation.cc:240] Negotiation successful
0927 14:51:58.231905 (+ 376us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":320,"mutex_wait_us":218,"spinlock_wait_cycles":2944,"thread_start_us":150,"threads_started":1}
I20260927 14:51:58.233343 7759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:51:58.215704 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:37119 (local address 127.0.98.3:33799)
0927 14:51:58.216748 (+ 1044us) negotiation.cc:107] Waiting for socket to connect
0927 14:51:58.216784 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0927 14:51:58.217267 (+ 483us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:51:58.220717 (+ 3450us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:51:58.220721 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:51:58.220933 (+ 212us) client_negotiation.cc:189] Negotiated authn=SASL
0927 14:51:58.221041 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:51:58.221046 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:51:58.221951 (+ 905us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:51:58.221957 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:51:58.222891 (+ 934us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:51:58.222901 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:51:58.223605 (+ 704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:51:58.223608 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:51:58.223703 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:51:58.224389 (+ 686us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 14:51:58.224411 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0927 14:51:58.226554 (+ 2143us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 14:51:58.228888 (+ 2334us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 14:51:58.228896 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 14:51:58.228916 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0927 14:51:58.229342 (+ 426us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 14:51:58.229670 (+ 328us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 14:51:58.229673 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 14:51:58.229675 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 14:51:58.229860 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 14:51:58.230434 (+ 574us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 14:51:58.230441 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 14:51:58.230772 (+ 331us) client_negotiation.cc:769] Sending connection context
0927 14:51:58.231866 (+ 1094us) client_negotiation.cc:240] Negotiation successful
0927 14:51:58.232028 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":956,"mutex_wait_us":185,"thread_start_us":122,"threads_started":1}
I20260927 14:51:58.234566 7755 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36527
I20260927 14:51:58.234855 7755 heartbeater.cc:464] Registering TS with master...
I20260927 14:51:58.235073 7765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:51:58.219688 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40382 (local address 127.0.98.62:36527)
0927 14:51:58.221388 (+ 1700us) server_negotiation.cc:206] Beginning negotiation
0927 14:51:58.221400 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 14:51:58.221412 (+ 12us) server_negotiation.cc:407] Connection header received
0927 14:51:58.221444 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:51:58.221447 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:51:58.221507 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:51:58.221600 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:51:58.222530 (+ 930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:51:58.223288 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:51:58.224560 (+ 1272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:51:58.224893 (+ 333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:51:58.224981 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:51:58.226753 (+ 1772us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:51:58.226757 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:51:58.226760 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 14:51:58.226786 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:51:58.229667 (+ 2881us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:51:58.232610 (+ 2943us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:51:58.232614 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:51:58.232615 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:51:58.232696 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:51:58.233806 (+ 1110us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:51:58.233810 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:51:58.233811 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:51:58.234053 (+ 242us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:51:58.234165 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0927 14:51:58.234772 (+ 607us) server_negotiation.cc:299] Negotiation successful
0927 14:51:58.234882 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1632,"thread_start_us":64,"threads_started":1}
I20260927 14:51:58.235438 7755 heartbeater.cc:511] Master 127.0.98.62:36527 requested a full tablet report, sending...
I20260927 14:51:58.236148 7760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:51:58.215703 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:38741 (local address 127.0.98.3:39895)
0927 14:51:58.216699 (+ 996us) negotiation.cc:107] Waiting for socket to connect
0927 14:51:58.216713 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 14:51:58.217264 (+ 551us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:51:58.218281 (+ 1017us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:51:58.218284 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:51:58.218949 (+ 665us) client_negotiation.cc:189] Negotiated authn=SASL
0927 14:51:58.219672 (+ 723us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:51:58.219692 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:51:58.220865 (+ 1173us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:51:58.220868 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:51:58.223310 (+ 2442us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:51:58.223318 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:51:58.227003 (+ 3685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:51:58.227007 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:51:58.227717 (+ 710us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:51:58.228786 (+ 1069us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 14:51:58.228800 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0927 14:51:58.230254 (+ 1454us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 14:51:58.233371 (+ 3117us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 14:51:58.233374 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 14:51:58.233376 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 14:51:58.233965 (+ 589us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 14:51:58.235013 (+ 1048us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 14:51:58.235016 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 14:51:58.235017 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 14:51:58.235097 (+ 80us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 14:51:58.235599 (+ 502us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 14:51:58.235602 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 14:51:58.235679 (+ 77us) client_negotiation.cc:769] Sending connection context
0927 14:51:58.235810 (+ 131us) client_negotiation.cc:240] Negotiation successful
0927 14:51:58.235938 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":914,"mutex_wait_us":170,"spinlock_wait_cycles":32384,"thread_start_us":130,"threads_started":1}
I20260927 14:51:58.236143 7484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:51:58.215284 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:39895 (local address 127.0.98.60:38741)
0927 14:51:58.215588 (+ 304us) server_negotiation.cc:206] Beginning negotiation
0927 14:51:58.215594 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 14:51:58.216809 (+ 1215us) server_negotiation.cc:407] Connection header received
0927 14:51:58.217486 (+ 677us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:51:58.217490 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:51:58.217546 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:51:58.217599 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:51:58.219848 (+ 2249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:51:58.220678 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:51:58.224578 (+ 3900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:51:58.224941 (+ 363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:51:58.225029 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:51:58.230408 (+ 5379us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:51:58.230411 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:51:58.230414 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 14:51:58.230442 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:51:58.232362 (+ 1920us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:51:58.234785 (+ 2423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:51:58.234788 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:51:58.234790 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:51:58.234889 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:51:58.235235 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:51:58.235238 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:51:58.235240 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:51:58.235436 (+ 196us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:51:58.235544 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0927 14:51:58.235862 (+ 318us) server_negotiation.cc:299] Negotiation successful
0927 14:51:58.236009 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":210}
I20260927 14:51:58.234563 7756 heartbeater.cc:347] Connected to a master server at 127.0.98.61:37119
I20260927 14:51:58.236246 7756 heartbeater.cc:464] Registering TS with master...
I20260927 14:51:58.236426 7756 heartbeater.cc:511] Master 127.0.98.61:37119 requested a full tablet report, sending...
I20260927 14:51:58.236862 7754 heartbeater.cc:347] Connected to a master server at 127.0.98.60:38741
I20260927 14:51:58.236943 7754 heartbeater.cc:464] Registering TS with master...
I20260927 14:51:58.237143 7754 heartbeater.cc:511] Master 127.0.98.60:38741 requested a full tablet report, sending...
I20260927 14:51:58.237221 6518 ts_manager.cc:194] Re-registered known tserver with Master: df8799b0da0c4b4e895000a21761211e (127.0.98.3:38555)
I20260927 14:51:58.237783 7301 ts_manager.cc:194] Registered new tserver with Master: df8799b0da0c4b4e895000a21761211e (127.0.98.3:38555)
I20260927 14:51:58.237950 6518 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:37479
I20260927 14:51:58.238246 7119 ts_manager.cc:194] Registered new tserver with Master: df8799b0da0c4b4e895000a21761211e (127.0.98.3:38555)
I20260927 14:51:58.241761 7768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:51:58.220048 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44658 (local address 127.0.98.60:38741)
0927 14:51:58.222005 (+ 1957us) server_negotiation.cc:206] Beginning negotiation
0927 14:51:58.222015 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 14:51:58.222027 (+ 12us) server_negotiation.cc:407] Connection header received
0927 14:51:58.222061 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:51:58.222064 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:51:58.222108 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:51:58.227241 (+ 5133us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:51:58.227364 (+ 123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:51:58.228463 (+ 1099us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:51:58.230730 (+ 2267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:51:58.231136 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:51:58.231703 (+ 567us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:51:58.237803 (+ 6100us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:51:58.237806 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:51:58.237809 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 14:51:58.237833 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:51:58.239573 (+ 1740us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:51:58.240116 (+ 543us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:51:58.240119 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:51:58.240122 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:51:58.240205 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:51:58.240579 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:51:58.240583 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:51:58.240585 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:51:58.240772 (+ 187us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:51:58.241419 (+ 647us) server_negotiation.cc:1035] Waiting for connection context
0927 14:51:58.241504 (+ 85us) server_negotiation.cc:299] Negotiation successful
0927 14:51:58.241612 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1888,"thread_start_us":91,"threads_started":1}
I20260927 14:51:58.242446 7769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:51:58.220214 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52214 (local address 127.0.98.61:37119)
0927 14:51:58.220424 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0927 14:51:58.220428 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 14:51:58.221866 (+ 1438us) server_negotiation.cc:407] Connection header received
0927 14:51:58.222336 (+ 470us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:51:58.222339 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:51:58.222394 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:51:58.222458 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:51:58.224945 (+ 2487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:51:58.225782 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:51:58.228663 (+ 2881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:51:58.229045 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:51:58.229142 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:51:58.237485 (+ 8343us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:51:58.237490 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:51:58.237495 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 14:51:58.237520 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:51:58.240464 (+ 2944us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:51:58.240859 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:51:58.240864 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:51:58.240866 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:51:58.240948 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:51:58.241245 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:51:58.241249 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:51:58.241251 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:51:58.241421 (+ 170us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:51:58.241553 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0927 14:51:58.241790 (+ 237us) server_negotiation.cc:299] Negotiation successful
0927 14:51:58.242291 (+ 501us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":144,"thread_start_us":83,"threads_started":1}
I20260927 14:51:58.239638 6518 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40382:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 14:51:58.243710 6518 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 14:51:58.246173 7077 sys_catalog.cc:455] T 00000000000000000000000000000000 P d60b79f337e343bca73eadfa1bd55f6a [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "d60b79f337e343bca73eadfa1bd55f6a" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "d60b79f337e343bca73eadfa1bd55f6a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36527 } } peers { permanent_uuid: "413c118097b64cd8af3711ec05bf636a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37119 } attrs { promote: false } } peers { permanent_uuid: "34720d7704db415f8c02e5d750103c4a" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 38741 } attrs { promote: true } } }
I20260927 14:51:58.246284 7077 sys_catalog.cc:458] T 00000000000000000000000000000000 P d60b79f337e343bca73eadfa1bd55f6a [sys.catalog]: This master's current role is: LEADER
I20260927 14:51:58.250146 7772 raft_consensus.cc:1050] T 00000000000000000000000000000000 P d60b79f337e343bca73eadfa1bd55f6a: attempting to promote NON_VOTER 34720d7704db415f8c02e5d750103c4a to VOTER
I20260927 14:51:58.250557 7772 consensus_queue.cc:237] T 00000000000000000000000000000000 P d60b79f337e343bca73eadfa1bd55f6a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "d60b79f337e343bca73eadfa1bd55f6a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36527 } } peers { permanent_uuid: "413c118097b64cd8af3711ec05bf636a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37119 } attrs { promote: false } } peers { permanent_uuid: "34720d7704db415f8c02e5d750103c4a" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38741 } attrs { promote: false } }
I20260927 14:51:58.251950 7306 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 34720d7704db415f8c02e5d750103c4a [term 1 LEARNER]: Refusing update from remote peer d60b79f337e343bca73eadfa1bd55f6a: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260927 14:51:58.252403 7077 consensus_queue.cc:1035] T 00000000000000000000000000000000 P d60b79f337e343bca73eadfa1bd55f6a [LEADER]: Connected to new peer: Peer: permanent_uuid: "34720d7704db415f8c02e5d750103c4a" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38741 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260927 14:51:58.252842 7129 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 413c118097b64cd8af3711ec05bf636a [term 1 FOLLOWER]: Refusing update from remote peer d60b79f337e343bca73eadfa1bd55f6a: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260927 14:51:58.253227 7077 consensus_queue.cc:1035] T 00000000000000000000000000000000 P d60b79f337e343bca73eadfa1bd55f6a [LEADER]: Connected to new peer: Peer: permanent_uuid: "413c118097b64cd8af3711ec05bf636a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37119 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260927 14:51:58.254626 7773 raft_consensus.cc:2932] T 00000000000000000000000000000000 P d60b79f337e343bca73eadfa1bd55f6a [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 7 to 9, 34720d7704db415f8c02e5d750103c4a (127.0.98.60) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "d60b79f337e343bca73eadfa1bd55f6a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36527 } } peers { permanent_uuid: "413c118097b64cd8af3711ec05bf636a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37119 } attrs { promote: false } } peers { permanent_uuid: "34720d7704db415f8c02e5d750103c4a" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38741 } attrs { promote: false } } }
I20260927 14:51:58.255151 7306 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 34720d7704db415f8c02e5d750103c4a [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, 34720d7704db415f8c02e5d750103c4a (127.0.98.60) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "d60b79f337e343bca73eadfa1bd55f6a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36527 } } peers { permanent_uuid: "413c118097b64cd8af3711ec05bf636a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37119 } attrs { promote: false } } peers { permanent_uuid: "34720d7704db415f8c02e5d750103c4a" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38741 } attrs { promote: false } } }
I20260927 14:51:58.256202 7077 sys_catalog.cc:455] T 00000000000000000000000000000000 P d60b79f337e343bca73eadfa1bd55f6a [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "d60b79f337e343bca73eadfa1bd55f6a" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "d60b79f337e343bca73eadfa1bd55f6a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36527 } } peers { permanent_uuid: "413c118097b64cd8af3711ec05bf636a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37119 } attrs { promote: false } } peers { permanent_uuid: "34720d7704db415f8c02e5d750103c4a" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38741 } attrs { promote: false } } }
I20260927 14:51:58.256310 7077 sys_catalog.cc:458] T 00000000000000000000000000000000 P d60b79f337e343bca73eadfa1bd55f6a [sys.catalog]: This master's current role is: LEADER
I20260927 14:51:58.256245 7773 sys_catalog.cc:455] T 00000000000000000000000000000000 P d60b79f337e343bca73eadfa1bd55f6a [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "d60b79f337e343bca73eadfa1bd55f6a" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "d60b79f337e343bca73eadfa1bd55f6a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36527 } } peers { permanent_uuid: "413c118097b64cd8af3711ec05bf636a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37119 } attrs { promote: false } } peers { permanent_uuid: "34720d7704db415f8c02e5d750103c4a" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38741 } attrs { promote: false } } }
I20260927 14:51:58.256470 7773 sys_catalog.cc:458] T 00000000000000000000000000000000 P d60b79f337e343bca73eadfa1bd55f6a [sys.catalog]: This master's current role is: LEADER
I20260927 14:51:58.257182 7343 sys_catalog.cc:455] T 00000000000000000000000000000000 P 34720d7704db415f8c02e5d750103c4a [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "d60b79f337e343bca73eadfa1bd55f6a" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "d60b79f337e343bca73eadfa1bd55f6a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36527 } } peers { permanent_uuid: "413c118097b64cd8af3711ec05bf636a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37119 } attrs { promote: false } } peers { permanent_uuid: "34720d7704db415f8c02e5d750103c4a" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38741 } attrs { promote: false } } }
I20260927 14:51:58.257328 7343 sys_catalog.cc:458] T 00000000000000000000000000000000 P 34720d7704db415f8c02e5d750103c4a [sys.catalog]: This master's current role is: FOLLOWER
I20260927 14:51:58.257658 7773 sys_catalog.cc:455] T 00000000000000000000000000000000 P d60b79f337e343bca73eadfa1bd55f6a [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "d60b79f337e343bca73eadfa1bd55f6a" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "d60b79f337e343bca73eadfa1bd55f6a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36527 } } peers { permanent_uuid: "413c118097b64cd8af3711ec05bf636a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37119 } attrs { promote: false } } peers { permanent_uuid: "34720d7704db415f8c02e5d750103c4a" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38741 } attrs { promote: false } } }
I20260927 14:51:58.257776 7773 sys_catalog.cc:458] T 00000000000000000000000000000000 P d60b79f337e343bca73eadfa1bd55f6a [sys.catalog]: This master's current role is: LEADER
I20260927 14:51:58.258044 7129 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 413c118097b64cd8af3711ec05bf636a [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, 34720d7704db415f8c02e5d750103c4a (127.0.98.60) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "d60b79f337e343bca73eadfa1bd55f6a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36527 } } peers { permanent_uuid: "413c118097b64cd8af3711ec05bf636a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37119 } attrs { promote: false } } peers { permanent_uuid: "34720d7704db415f8c02e5d750103c4a" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38741 } attrs { promote: false } } }
I20260927 14:51:58.261765 7781 sys_catalog.cc:455] T 00000000000000000000000000000000 P 413c118097b64cd8af3711ec05bf636a [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "d60b79f337e343bca73eadfa1bd55f6a" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "d60b79f337e343bca73eadfa1bd55f6a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36527 } } peers { permanent_uuid: "413c118097b64cd8af3711ec05bf636a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37119 } attrs { promote: false } } peers { permanent_uuid: "34720d7704db415f8c02e5d750103c4a" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38741 } attrs { promote: false } } }
I20260927 14:51:58.261902 7781 sys_catalog.cc:458] T 00000000000000000000000000000000 P 413c118097b64cd8af3711ec05bf636a [sys.catalog]: This master's current role is: FOLLOWER
I20260927 14:51:58.265290 7782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:51:58.260062 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:43017 (local address 127.0.0.1:33940)
0927 14:51:58.260916 (+ 854us) negotiation.cc:107] Waiting for socket to connect
0927 14:51:58.260925 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0927 14:51:58.260982 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:51:58.261476 (+ 494us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:51:58.261479 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:51:58.261496 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 14:51:58.261604 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:51:58.261611 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:51:58.262755 (+ 1144us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:51:58.262759 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:51:58.263904 (+ 1145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:51:58.263914 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:51:58.264858 (+ 944us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:51:58.264862 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:51:58.264952 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:51:58.264974 (+ 22us) client_negotiation.cc:769] Sending connection context
0927 14:51:58.265037 (+ 63us) client_negotiation.cc:240] Negotiation successful
0927 14:51:58.265141 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":815,"thread_start_us":93,"threads_started":1}
I20260927 14:51:58.265362 7786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:51:58.260204 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33940 (local address 127.0.98.1:43017)
0927 14:51:58.260528 (+ 324us) server_negotiation.cc:206] Beginning negotiation
0927 14:51:58.260541 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 14:51:58.261001 (+ 460us) server_negotiation.cc:407] Connection header received
0927 14:51:58.261136 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:51:58.261143 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:51:58.261332 (+ 189us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:51:58.261504 (+ 172us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 14:51:58.261730 (+ 226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:51:58.262631 (+ 901us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:51:58.264042 (+ 1411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:51:58.264724 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:51:58.264860 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:51:58.265019 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0927 14:51:58.265107 (+ 88us) server_negotiation.cc:299] Negotiation successful
0927 14:51:58.265220 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":217,"thread_start_us":99,"threads_started":1}
I20260927 14:51:58.265662 7644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:51:58.259374 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:34621 (local address 127.0.0.1:40198)
0927 14:51:58.259683 (+ 309us) negotiation.cc:107] Waiting for socket to connect
0927 14:51:58.259703 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0927 14:51:58.259782 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:51:58.261685 (+ 1903us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:51:58.261688 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:51:58.261708 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 14:51:58.261779 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:51:58.261784 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:51:58.263098 (+ 1314us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:51:58.263102 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:51:58.264208 (+ 1106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:51:58.264216 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:51:58.265287 (+ 1071us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:51:58.265291 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:51:58.265375 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:51:58.265398 (+ 23us) client_negotiation.cc:769] Sending connection context
0927 14:51:58.265446 (+ 48us) client_negotiation.cc:240] Negotiation successful
0927 14:51:58.265526 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":231}
I20260927 14:51:58.266079 7785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:51:58.259629 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:38555 (local address 127.0.0.1:32878)
0927 14:51:58.260698 (+ 1069us) negotiation.cc:107] Waiting for socket to connect
0927 14:51:58.260711 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 14:51:58.260791 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:51:58.262055 (+ 1264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:51:58.262058 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:51:58.262084 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 14:51:58.262168 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:51:58.262173 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:51:58.263577 (+ 1404us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:51:58.263580 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:51:58.264598 (+ 1018us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:51:58.264607 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:51:58.265662 (+ 1055us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:51:58.265667 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:51:58.265777 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:51:58.265799 (+ 22us) client_negotiation.cc:769] Sending connection context
0927 14:51:58.265880 (+ 81us) client_negotiation.cc:240] Negotiation successful
0927 14:51:58.265965 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1012,"mutex_wait_us":3,"thread_start_us":95,"threads_started":1}
I20260927 14:51:58.266130 7784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:51:58.260016 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32878 (local address 127.0.98.3:38555)
0927 14:51:58.261267 (+ 1251us) server_negotiation.cc:206] Beginning negotiation
0927 14:51:58.261278 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 14:51:58.261305 (+ 27us) server_negotiation.cc:407] Connection header received
0927 14:51:58.261364 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:51:58.261368 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:51:58.261557 (+ 189us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:51:58.262317 (+ 760us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 14:51:58.262439 (+ 122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:51:58.263432 (+ 993us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:51:58.264739 (+ 1307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:51:58.265543 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:51:58.265643 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:51:58.265779 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0927 14:51:58.265899 (+ 120us) server_negotiation.cc:299] Negotiation successful
0927 14:51:58.265990 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1160,"thread_start_us":83,"threads_started":1}
I20260927 14:51:58.266377 7783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:51:58.259958 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40198 (local address 127.0.98.2:34621)
0927 14:51:58.261243 (+ 1285us) server_negotiation.cc:206] Beginning negotiation
0927 14:51:58.261255 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 14:51:58.261278 (+ 23us) server_negotiation.cc:407] Connection header received
0927 14:51:58.261357 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:51:58.261363 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:51:58.261557 (+ 194us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:51:58.261701 (+ 144us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 14:51:58.261866 (+ 165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:51:58.262981 (+ 1115us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:51:58.264326 (+ 1345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:51:58.265146 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:51:58.265973 (+ 827us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:51:58.266102 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0927 14:51:58.266152 (+ 50us) server_negotiation.cc:299] Negotiation successful
0927 14:51:58.266244 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1185,"thread_start_us":129,"threads_started":1}
I20260927 14:51:58.268724 7551 tablet_service.cc:1467] Processing CreateTablet for tablet 22cf03c91920459998c9762b04d634ba (DEFAULT_TABLE table=test-table [id=7fcbee22dc1549e287d8901ad89b1738]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 14:51:58.268743 7688 tablet_service.cc:1467] Processing CreateTablet for tablet 22cf03c91920459998c9762b04d634ba (DEFAULT_TABLE table=test-table [id=7fcbee22dc1549e287d8901ad89b1738]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 14:51:58.268743 7414 tablet_service.cc:1467] Processing CreateTablet for tablet 22cf03c91920459998c9762b04d634ba (DEFAULT_TABLE table=test-table [id=7fcbee22dc1549e287d8901ad89b1738]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 14:51:58.269701 7551 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 22cf03c91920459998c9762b04d634ba. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 14:51:58.269721 7688 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 22cf03c91920459998c9762b04d634ba. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 14:51:58.269721 7414 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 22cf03c91920459998c9762b04d634ba. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 14:51:58.275996 7787 tablet_bootstrap.cc:492] T 22cf03c91920459998c9762b04d634ba P a473db7856d24a8ba3efb9bbc9e12de9: Bootstrap starting.
I20260927 14:51:58.277405 7788 tablet_bootstrap.cc:492] T 22cf03c91920459998c9762b04d634ba P f37a65090f8b4b8f9a4facdb930c9eec: Bootstrap starting.
I20260927 14:51:58.279065 7787 tablet_bootstrap.cc:654] T 22cf03c91920459998c9762b04d634ba P a473db7856d24a8ba3efb9bbc9e12de9: Neither blocks nor log segments found. Creating new log.
I20260927 14:51:58.279093 7789 tablet_bootstrap.cc:492] T 22cf03c91920459998c9762b04d634ba P df8799b0da0c4b4e895000a21761211e: Bootstrap starting.
I20260927 14:51:58.279860 7788 tablet_bootstrap.cc:654] T 22cf03c91920459998c9762b04d634ba P f37a65090f8b4b8f9a4facdb930c9eec: Neither blocks nor log segments found. Creating new log.
I20260927 14:51:58.279891 7787 log.cc:824] T 22cf03c91920459998c9762b04d634ba P a473db7856d24a8ba3efb9bbc9e12de9: Log is configured to *not* fsync() on all Append() calls
I20260927 14:51:58.280588 7788 log.cc:824] T 22cf03c91920459998c9762b04d634ba P f37a65090f8b4b8f9a4facdb930c9eec: Log is configured to *not* fsync() on all Append() calls
I20260927 14:51:58.280815 7789 tablet_bootstrap.cc:654] T 22cf03c91920459998c9762b04d634ba P df8799b0da0c4b4e895000a21761211e: Neither blocks nor log segments found. Creating new log.
I20260927 14:51:58.281581 7789 log.cc:824] T 22cf03c91920459998c9762b04d634ba P df8799b0da0c4b4e895000a21761211e: Log is configured to *not* fsync() on all Append() calls
I20260927 14:51:58.281829 7787 tablet_bootstrap.cc:492] T 22cf03c91920459998c9762b04d634ba P a473db7856d24a8ba3efb9bbc9e12de9: No bootstrap required, opened a new log
I20260927 14:51:58.282019 7787 ts_tablet_manager.cc:1397] T 22cf03c91920459998c9762b04d634ba P a473db7856d24a8ba3efb9bbc9e12de9: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260927 14:51:58.282442 7788 tablet_bootstrap.cc:492] T 22cf03c91920459998c9762b04d634ba P f37a65090f8b4b8f9a4facdb930c9eec: No bootstrap required, opened a new log
I20260927 14:51:58.282598 7788 ts_tablet_manager.cc:1397] T 22cf03c91920459998c9762b04d634ba P f37a65090f8b4b8f9a4facdb930c9eec: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260927 14:51:58.283399 7789 tablet_bootstrap.cc:492] T 22cf03c91920459998c9762b04d634ba P df8799b0da0c4b4e895000a21761211e: No bootstrap required, opened a new log
I20260927 14:51:58.283572 7789 ts_tablet_manager.cc:1397] T 22cf03c91920459998c9762b04d634ba P df8799b0da0c4b4e895000a21761211e: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260927 14:51:58.285769 7787 raft_consensus.cc:348] T 22cf03c91920459998c9762b04d634ba P a473db7856d24a8ba3efb9bbc9e12de9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a473db7856d24a8ba3efb9bbc9e12de9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34621 } } peers { permanent_uuid: "df8799b0da0c4b4e895000a21761211e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38555 } } peers { permanent_uuid: "f37a65090f8b4b8f9a4facdb930c9eec" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43017 } }
I20260927 14:51:58.286044 7787 raft_consensus.cc:374] T 22cf03c91920459998c9762b04d634ba P a473db7856d24a8ba3efb9bbc9e12de9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 14:51:58.286103 7787 raft_consensus.cc:729] T 22cf03c91920459998c9762b04d634ba P a473db7856d24a8ba3efb9bbc9e12de9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a473db7856d24a8ba3efb9bbc9e12de9, State: Initialized, Role: FOLLOWER
I20260927 14:51:58.286535 7787 consensus_queue.cc:260] T 22cf03c91920459998c9762b04d634ba P a473db7856d24a8ba3efb9bbc9e12de9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a473db7856d24a8ba3efb9bbc9e12de9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34621 } } peers { permanent_uuid: "df8799b0da0c4b4e895000a21761211e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38555 } } peers { permanent_uuid: "f37a65090f8b4b8f9a4facdb930c9eec" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43017 } }
I20260927 14:51:58.287521 7787 ts_tablet_manager.cc:1428] T 22cf03c91920459998c9762b04d634ba P a473db7856d24a8ba3efb9bbc9e12de9: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260927 14:51:58.287734 7618 heartbeater.cc:503] Master 127.0.98.62:36527 was elected leader, sending a full tablet report...
I20260927 14:51:58.287801 7789 raft_consensus.cc:348] T 22cf03c91920459998c9762b04d634ba P df8799b0da0c4b4e895000a21761211e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a473db7856d24a8ba3efb9bbc9e12de9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34621 } } peers { permanent_uuid: "df8799b0da0c4b4e895000a21761211e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38555 } } peers { permanent_uuid: "f37a65090f8b4b8f9a4facdb930c9eec" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43017 } }
I20260927 14:51:58.288036 7789 raft_consensus.cc:374] T 22cf03c91920459998c9762b04d634ba P df8799b0da0c4b4e895000a21761211e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 14:51:58.288089 7789 raft_consensus.cc:729] T 22cf03c91920459998c9762b04d634ba P df8799b0da0c4b4e895000a21761211e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df8799b0da0c4b4e895000a21761211e, State: Initialized, Role: FOLLOWER
I20260927 14:51:58.287930 7788 raft_consensus.cc:348] T 22cf03c91920459998c9762b04d634ba P f37a65090f8b4b8f9a4facdb930c9eec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a473db7856d24a8ba3efb9bbc9e12de9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34621 } } peers { permanent_uuid: "df8799b0da0c4b4e895000a21761211e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38555 } } peers { permanent_uuid: "f37a65090f8b4b8f9a4facdb930c9eec" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43017 } }
I20260927 14:51:58.288153 7788 raft_consensus.cc:374] T 22cf03c91920459998c9762b04d634ba P f37a65090f8b4b8f9a4facdb930c9eec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 14:51:58.288204 7788 raft_consensus.cc:729] T 22cf03c91920459998c9762b04d634ba P f37a65090f8b4b8f9a4facdb930c9eec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f37a65090f8b4b8f9a4facdb930c9eec, State: Initialized, Role: FOLLOWER
I20260927 14:51:58.288599 7788 consensus_queue.cc:260] T 22cf03c91920459998c9762b04d634ba P f37a65090f8b4b8f9a4facdb930c9eec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a473db7856d24a8ba3efb9bbc9e12de9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34621 } } peers { permanent_uuid: "df8799b0da0c4b4e895000a21761211e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38555 } } peers { permanent_uuid: "f37a65090f8b4b8f9a4facdb930c9eec" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43017 } }
I20260927 14:51:58.289553 7788 ts_tablet_manager.cc:1428] T 22cf03c91920459998c9762b04d634ba P f37a65090f8b4b8f9a4facdb930c9eec: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260927 14:51:58.290663 7481 heartbeater.cc:503] Master 127.0.98.62:36527 was elected leader, sending a full tablet report...
I20260927 14:51:58.291441 7789 consensus_queue.cc:260] T 22cf03c91920459998c9762b04d634ba P df8799b0da0c4b4e895000a21761211e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a473db7856d24a8ba3efb9bbc9e12de9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34621 } } peers { permanent_uuid: "df8799b0da0c4b4e895000a21761211e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38555 } } peers { permanent_uuid: "f37a65090f8b4b8f9a4facdb930c9eec" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43017 } }
I20260927 14:51:58.292474 7789 ts_tablet_manager.cc:1428] T 22cf03c91920459998c9762b04d634ba P df8799b0da0c4b4e895000a21761211e: Time spent starting tablet: real 0.009s user 0.006s sys 0.000s
I20260927 14:51:58.293169 7755 heartbeater.cc:503] Master 127.0.98.62:36527 was elected leader, sending a full tablet report...
W20260927 14:51:58.325729 7483 tablet.cc:2369] T 22cf03c91920459998c9762b04d634ba P f37a65090f8b4b8f9a4facdb930c9eec: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 14:51:58.375850 7795 raft_consensus.cc:482] T 22cf03c91920459998c9762b04d634ba P df8799b0da0c4b4e895000a21761211e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 14:51:58.376107 7795 raft_consensus.cc:504] T 22cf03c91920459998c9762b04d634ba P df8799b0da0c4b4e895000a21761211e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a473db7856d24a8ba3efb9bbc9e12de9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34621 } } peers { permanent_uuid: "df8799b0da0c4b4e895000a21761211e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38555 } } peers { permanent_uuid: "f37a65090f8b4b8f9a4facdb930c9eec" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43017 } }
I20260927 14:51:58.377260 7795 leader_election.cc:288] T 22cf03c91920459998c9762b04d634ba P df8799b0da0c4b4e895000a21761211e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a473db7856d24a8ba3efb9bbc9e12de9 (127.0.98.2:34621), f37a65090f8b4b8f9a4facdb930c9eec (127.0.98.1:43017)
I20260927 14:51:58.382068 7760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:51:58.377572 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:34621 (local address 127.0.98.3:48307)
0927 14:51:58.377715 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0927 14:51:58.377732 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 14:51:58.377836 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:51:58.378220 (+ 384us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:51:58.378223 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:51:58.378243 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 14:51:58.378373 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:51:58.378379 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:51:58.379454 (+ 1075us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:51:58.379457 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:51:58.380700 (+ 1243us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:51:58.380711 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:51:58.381617 (+ 906us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:51:58.381620 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:51:58.381788 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:51:58.381818 (+ 30us) client_negotiation.cc:769] Sending connection context
0927 14:51:58.381879 (+ 61us) client_negotiation.cc:240] Negotiation successful
0927 14:51:58.381924 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":55,"mutex_wait_us":38}
I20260927 14:51:58.382167 7786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:51:58.378047 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:49131 (local address 127.0.98.1:43017)
0927 14:51:58.378162 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0927 14:51:58.378165 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 14:51:58.378641 (+ 476us) server_negotiation.cc:407] Connection header received
0927 14:51:58.378702 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:51:58.378705 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:51:58.378759 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:51:58.378849 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 14:51:58.379118 (+ 269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:51:58.379685 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:51:58.380696 (+ 1011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:51:58.381326 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:51:58.381444 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:51:58.381525 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0927 14:51:58.381984 (+ 459us) server_negotiation.cc:299] Negotiation successful
0927 14:51:58.382053 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260927 14:51:58.382539 7783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:51:58.377617 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:48307 (local address 127.0.98.2:34621)
0927 14:51:58.377763 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0927 14:51:58.377767 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 14:51:58.377792 (+ 25us) server_negotiation.cc:407] Connection header received
0927 14:51:58.377991 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:51:58.377995 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:51:58.378046 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:51:58.378110 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 14:51:58.378508 (+ 398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:51:58.379346 (+ 838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:51:58.380821 (+ 1475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:51:58.381434 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:51:58.381515 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:51:58.381575 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0927 14:51:58.382330 (+ 755us) server_negotiation.cc:299] Negotiation successful
0927 14:51:58.382422 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260927 14:51:58.382916 7758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:51:58.377969 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:43017 (local address 127.0.98.3:49131)
0927 14:51:58.378553 (+ 584us) negotiation.cc:107] Waiting for socket to connect
0927 14:51:58.378565 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 14:51:58.378628 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:51:58.378876 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:51:58.378878 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:51:58.378895 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 14:51:58.379001 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:51:58.379007 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:51:58.379814 (+ 807us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:51:58.379818 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:51:58.380578 (+ 760us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:51:58.380587 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:51:58.381582 (+ 995us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:51:58.381588 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:51:58.381738 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:51:58.381772 (+ 34us) client_negotiation.cc:769] Sending connection context
0927 14:51:58.381847 (+ 75us) client_negotiation.cc:240] Negotiation successful
0927 14:51:58.381922 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":523,"mutex_wait_us":28}
I20260927 14:51:58.383121 7571 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "22cf03c91920459998c9762b04d634ba" candidate_uuid: "df8799b0da0c4b4e895000a21761211e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a473db7856d24a8ba3efb9bbc9e12de9" is_pre_election: true
I20260927 14:51:58.383424 7571 raft_consensus.cc:2454] T 22cf03c91920459998c9762b04d634ba P a473db7856d24a8ba3efb9bbc9e12de9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df8799b0da0c4b4e895000a21761211e in term 0.
I20260927 14:51:58.383518 7434 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "22cf03c91920459998c9762b04d634ba" candidate_uuid: "df8799b0da0c4b4e895000a21761211e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f37a65090f8b4b8f9a4facdb930c9eec" is_pre_election: true
I20260927 14:51:58.383845 7434 raft_consensus.cc:2454] T 22cf03c91920459998c9762b04d634ba P f37a65090f8b4b8f9a4facdb930c9eec [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df8799b0da0c4b4e895000a21761211e in term 0.
I20260927 14:51:58.383905 7642 leader_election.cc:302] T 22cf03c91920459998c9762b04d634ba P df8799b0da0c4b4e895000a21761211e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a473db7856d24a8ba3efb9bbc9e12de9, df8799b0da0c4b4e895000a21761211e; no voters:
I20260927 14:51:58.384377 7795 raft_consensus.cc:2781] T 22cf03c91920459998c9762b04d634ba P df8799b0da0c4b4e895000a21761211e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 14:51:58.384442 7795 raft_consensus.cc:482] T 22cf03c91920459998c9762b04d634ba P df8799b0da0c4b4e895000a21761211e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 14:51:58.384539 7795 raft_consensus.cc:3037] T 22cf03c91920459998c9762b04d634ba P df8799b0da0c4b4e895000a21761211e [term 0 FOLLOWER]: Advancing to term 1
I20260927 14:51:58.386308 7795 raft_consensus.cc:504] T 22cf03c91920459998c9762b04d634ba P df8799b0da0c4b4e895000a21761211e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a473db7856d24a8ba3efb9bbc9e12de9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34621 } } peers { permanent_uuid: "df8799b0da0c4b4e895000a21761211e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38555 } } peers { permanent_uuid: "f37a65090f8b4b8f9a4facdb930c9eec" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43017 } }
I20260927 14:51:58.386673 7795 leader_election.cc:288] T 22cf03c91920459998c9762b04d634ba P df8799b0da0c4b4e895000a21761211e [CANDIDATE]: Term 1 election: Requested vote from peers a473db7856d24a8ba3efb9bbc9e12de9 (127.0.98.2:34621), f37a65090f8b4b8f9a4facdb930c9eec (127.0.98.1:43017)
I20260927 14:51:58.386976 7571 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "22cf03c91920459998c9762b04d634ba" candidate_uuid: "df8799b0da0c4b4e895000a21761211e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a473db7856d24a8ba3efb9bbc9e12de9"
I20260927 14:51:58.387112 7571 raft_consensus.cc:3037] T 22cf03c91920459998c9762b04d634ba P a473db7856d24a8ba3efb9bbc9e12de9 [term 0 FOLLOWER]: Advancing to term 1
I20260927 14:51:58.388118 7571 raft_consensus.cc:2454] T 22cf03c91920459998c9762b04d634ba P a473db7856d24a8ba3efb9bbc9e12de9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df8799b0da0c4b4e895000a21761211e in term 1.
I20260927 14:51:58.388373 7642 leader_election.cc:302] T 22cf03c91920459998c9762b04d634ba P df8799b0da0c4b4e895000a21761211e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a473db7856d24a8ba3efb9bbc9e12de9, df8799b0da0c4b4e895000a21761211e; no voters:
I20260927 14:51:58.388370 7434 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "22cf03c91920459998c9762b04d634ba" candidate_uuid: "df8799b0da0c4b4e895000a21761211e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f37a65090f8b4b8f9a4facdb930c9eec"
I20260927 14:51:58.388489 7434 raft_consensus.cc:3037] T 22cf03c91920459998c9762b04d634ba P f37a65090f8b4b8f9a4facdb930c9eec [term 0 FOLLOWER]: Advancing to term 1
I20260927 14:51:58.388536 7795 raft_consensus.cc:2781] T 22cf03c91920459998c9762b04d634ba P df8799b0da0c4b4e895000a21761211e [term 1 FOLLOWER]: Leader election won for term 1
I20260927 14:51:58.388754 7795 raft_consensus.cc:686] T 22cf03c91920459998c9762b04d634ba P df8799b0da0c4b4e895000a21761211e [term 1 LEADER]: Becoming Leader. State: Replica: df8799b0da0c4b4e895000a21761211e, State: Running, Role: LEADER
I20260927 14:51:58.389495 7795 consensus_queue.cc:237] T 22cf03c91920459998c9762b04d634ba P df8799b0da0c4b4e895000a21761211e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a473db7856d24a8ba3efb9bbc9e12de9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34621 } } peers { permanent_uuid: "df8799b0da0c4b4e895000a21761211e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38555 } } peers { permanent_uuid: "f37a65090f8b4b8f9a4facdb930c9eec" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43017 } }
I20260927 14:51:58.389693 7434 raft_consensus.cc:2454] T 22cf03c91920459998c9762b04d634ba P f37a65090f8b4b8f9a4facdb930c9eec [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df8799b0da0c4b4e895000a21761211e in term 1.
I20260927 14:51:58.392930 6518 catalog_manager.cc:5495] T 22cf03c91920459998c9762b04d634ba P df8799b0da0c4b4e895000a21761211e reported cstate change: term changed from 0 to 1, leader changed from <none> to df8799b0da0c4b4e895000a21761211e (127.0.98.3). New cstate: current_term: 1 leader_uuid: "df8799b0da0c4b4e895000a21761211e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a473db7856d24a8ba3efb9bbc9e12de9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34621 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "df8799b0da0c4b4e895000a21761211e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38555 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f37a65090f8b4b8f9a4facdb930c9eec" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43017 } health_report { overall_health: UNKNOWN } } }
I20260927 14:51:58.419826 7784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:51:58.416194 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32880 (local address 127.0.98.3:38555)
0927 14:51:58.416318 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0927 14:51:58.416321 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 14:51:58.416336 (+ 15us) server_negotiation.cc:407] Connection header received
0927 14:51:58.416411 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:51:58.416413 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:51:58.416471 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:51:58.416569 (+ 98us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 14:51:58.416811 (+ 242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:51:58.417383 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:51:58.418223 (+ 840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:51:58.418481 (+ 258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:51:58.418563 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:51:58.419006 (+ 443us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 14:51:58.419438 (+ 432us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 14:51:58.419574 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0927 14:51:58.419610 (+ 36us) server_negotiation.cc:299] Negotiation successful
0927 14:51:58.419666 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260927 14:51:58.425714 7571 raft_consensus.cc:1261] T 22cf03c91920459998c9762b04d634ba P a473db7856d24a8ba3efb9bbc9e12de9 [term 1 FOLLOWER]: Refusing update from remote peer df8799b0da0c4b4e895000a21761211e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 14:51:58.425714 7434 raft_consensus.cc:1261] T 22cf03c91920459998c9762b04d634ba P f37a65090f8b4b8f9a4facdb930c9eec [term 1 FOLLOWER]: Refusing update from remote peer df8799b0da0c4b4e895000a21761211e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 14:51:58.426468 7795 consensus_queue.cc:1035] T 22cf03c91920459998c9762b04d634ba P df8799b0da0c4b4e895000a21761211e [LEADER]: Connected to new peer: Peer: permanent_uuid: "f37a65090f8b4b8f9a4facdb930c9eec" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43017 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 14:51:58.426684 7796 consensus_queue.cc:1035] T 22cf03c91920459998c9762b04d634ba P df8799b0da0c4b4e895000a21761211e [LEADER]: Connected to new peer: Peer: permanent_uuid: "a473db7856d24a8ba3efb9bbc9e12de9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34621 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 14:51:58.433209 7799 mvcc.cc:205] Tried to move back new op lower bound from 7333972862664073216 to 7333972862526099456. Current Snapshot: MvccSnapshot[applied={T|T < 7333972862664073216}]
I20260927 14:51:58.436384 7800 mvcc.cc:205] Tried to move back new op lower bound from 7333972862664073216 to 7333972862526099456. Current Snapshot: MvccSnapshot[applied={T|T < 7333972862664073216}]
I20260927 14:51:58.441646 7801 mvcc.cc:205] Tried to move back new op lower bound from 7333972862664073216 to 7333972862526099456. Current Snapshot: MvccSnapshot[applied={T|T < 7333972862664073216 or (T in {7333972862664073216})}]
I20260927 14:51:58.446416 6517 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40382:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 14:51:58.446588 6517 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40382:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 14:51:58.450611 6517 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d60b79f337e343bca73eadfa1bd55f6a: Sending DeleteTablet for 3 replicas of tablet 22cf03c91920459998c9762b04d634ba
I20260927 14:51:58.451602 7688 tablet_service.cc:1514] Processing DeleteTablet for tablet 22cf03c91920459998c9762b04d634ba with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 14:51:58 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:32878
I20260927 14:51:58.451714 7414 tablet_service.cc:1514] Processing DeleteTablet for tablet 22cf03c91920459998c9762b04d634ba with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 14:51:58 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:33940
I20260927 14:51:58.452328 7551 tablet_service.cc:1514] Processing DeleteTablet for tablet 22cf03c91920459998c9762b04d634ba with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 14:51:58 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:40198
I20260927 14:51:58.452816 7809 tablet_replica.cc:330] T 22cf03c91920459998c9762b04d634ba P f37a65090f8b4b8f9a4facdb930c9eec: stopping tablet replica
I20260927 14:51:58.453208 7809 raft_consensus.cc:2229] T 22cf03c91920459998c9762b04d634ba P f37a65090f8b4b8f9a4facdb930c9eec [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 14:51:58.453521 7809 raft_consensus.cc:2258] T 22cf03c91920459998c9762b04d634ba P f37a65090f8b4b8f9a4facdb930c9eec [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 14:51:58.453910 7810 tablet_replica.cc:330] T 22cf03c91920459998c9762b04d634ba P a473db7856d24a8ba3efb9bbc9e12de9: stopping tablet replica
I20260927 14:51:58.454187 7810 raft_consensus.cc:2229] T 22cf03c91920459998c9762b04d634ba P a473db7856d24a8ba3efb9bbc9e12de9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 14:51:58.454457 7810 raft_consensus.cc:2258] T 22cf03c91920459998c9762b04d634ba P a473db7856d24a8ba3efb9bbc9e12de9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 14:51:58.454771 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6936m5/build/debug/bin/kudu with pid 6486
I20260927 14:51:58.456265 7809 ts_tablet_manager.cc:1905] T 22cf03c91920459998c9762b04d634ba P f37a65090f8b4b8f9a4facdb930c9eec: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 14:51:58.456543 7810 ts_tablet_manager.cc:1905] T 22cf03c91920459998c9762b04d634ba P a473db7856d24a8ba3efb9bbc9e12de9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 14:51:58.457667 7808 tablet_replica.cc:330] T 22cf03c91920459998c9762b04d634ba P df8799b0da0c4b4e895000a21761211e: stopping tablet replica
I20260927 14:51:58.458186 7808 raft_consensus.cc:2229] T 22cf03c91920459998c9762b04d634ba P df8799b0da0c4b4e895000a21761211e [term 1 LEADER]: Raft consensus shutting down.
I20260927 14:51:58.458621 7808 raft_consensus.cc:2258] T 22cf03c91920459998c9762b04d634ba P df8799b0da0c4b4e895000a21761211e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 14:51:58.459052 7809 ts_tablet_manager.cc:1918] T 22cf03c91920459998c9762b04d634ba P f37a65090f8b4b8f9a4facdb930c9eec: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 14:51:58.459142 7809 log.cc:1196] T 22cf03c91920459998c9762b04d634ba P f37a65090f8b4b8f9a4facdb930c9eec: Deleting WAL directory at /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790520684602251-392-0/minicluster-data/ts-0/wal/wals/22cf03c91920459998c9762b04d634ba
I20260927 14:51:58.459475 7809 ts_tablet_manager.cc:1939] T 22cf03c91920459998c9762b04d634ba P f37a65090f8b4b8f9a4facdb930c9eec: Deleting consensus metadata
I20260927 14:51:58.460526 7810 ts_tablet_manager.cc:1918] T 22cf03c91920459998c9762b04d634ba P a473db7856d24a8ba3efb9bbc9e12de9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 14:51:58.460606 7810 log.cc:1196] T 22cf03c91920459998c9762b04d634ba P a473db7856d24a8ba3efb9bbc9e12de9: Deleting WAL directory at /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790520684602251-392-0/minicluster-data/ts-1/wal/wals/22cf03c91920459998c9762b04d634ba
I20260927 14:51:58.460916 7810 ts_tablet_manager.cc:1939] T 22cf03c91920459998c9762b04d634ba P a473db7856d24a8ba3efb9bbc9e12de9: Deleting consensus metadata
I20260927 14:51:58.462332 7808 ts_tablet_manager.cc:1905] T 22cf03c91920459998c9762b04d634ba P df8799b0da0c4b4e895000a21761211e: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260927 14:51:58.462646 7367 connection.cc:535] server connection from 127.0.0.1:33940 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260927 14:51:58.463042 7504 connection.cc:535] server connection from 127.0.0.1:40198 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260927 14:51:58.465348 7808 ts_tablet_manager.cc:1918] T 22cf03c91920459998c9762b04d634ba P df8799b0da0c4b4e895000a21761211e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 14:51:58.465453 7808 log.cc:1196] T 22cf03c91920459998c9762b04d634ba P df8799b0da0c4b4e895000a21761211e: Deleting WAL directory at /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790520684602251-392-0/minicluster-data/ts-2/wal/wals/22cf03c91920459998c9762b04d634ba
I20260927 14:51:58.465724 7808 ts_tablet_manager.cc:1939] T 22cf03c91920459998c9762b04d634ba P df8799b0da0c4b4e895000a21761211e: Deleting consensus metadata
W20260927 14:51:58.466748 7641 connection.cc:441] server connection from 127.0.0.1:32878 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:32878 (request call id 1) could send its response
I20260927 14:51:58.472106 7769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:51:58.453014 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52218 (local address 127.0.98.61:37119)
0927 14:51:58.453144 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0927 14:51:58.453150 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 14:51:58.454106 (+ 956us) server_negotiation.cc:407] Connection header received
0927 14:51:58.454276 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:51:58.454279 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:51:58.454325 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:51:58.454371 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:51:58.456415 (+ 2044us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:51:58.457352 (+ 937us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:51:58.458883 (+ 1531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:51:58.459297 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:51:58.459392 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:51:58.467846 (+ 8454us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:51:58.467852 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:51:58.467856 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 14:51:58.467883 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:51:58.469698 (+ 1815us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:51:58.470188 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:51:58.470191 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:51:58.470193 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:51:58.470274 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:51:58.470545 (+ 271us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:51:58.470548 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:51:58.470549 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:51:58.470759 (+ 210us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:51:58.470864 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0927 14:51:58.471450 (+ 586us) server_negotiation.cc:299] Negotiation successful
0927 14:51:58.471879 (+ 429us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260927 14:51:58.476076 7241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:51:58.466366 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52226 (local address 127.0.98.61:37119)
0927 14:51:58.466470 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0927 14:51:58.466475 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 14:51:58.467128 (+ 653us) server_negotiation.cc:407] Connection header received
0927 14:51:58.467260 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:51:58.467263 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:51:58.467318 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:51:58.467396 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:51:58.468226 (+ 830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:51:58.469009 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:51:58.470262 (+ 1253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:51:58.470579 (+ 317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:51:58.470653 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:51:58.472188 (+ 1535us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:51:58.472194 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:51:58.472199 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 14:51:58.472223 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:51:58.473755 (+ 1532us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:51:58.474719 (+ 964us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:51:58.474722 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:51:58.474724 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:51:58.474812 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:51:58.475098 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:51:58.475101 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:51:58.475103 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:51:58.475336 (+ 233us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:51:58.475447 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0927 14:51:58.475827 (+ 380us) server_negotiation.cc:299] Negotiation successful
0927 14:51:58.475930 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260927 14:51:58.477593 7484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:51:58.466749 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44674 (local address 127.0.98.60:38741)
0927 14:51:58.466860 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0927 14:51:58.466865 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 14:51:58.467481 (+ 616us) server_negotiation.cc:407] Connection header received
0927 14:51:58.467652 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:51:58.467655 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:51:58.467705 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:51:58.467759 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:51:58.468650 (+ 891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:51:58.469556 (+ 906us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:51:58.470768 (+ 1212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:51:58.471078 (+ 310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:51:58.471159 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:51:58.473402 (+ 2243us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:51:58.473406 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:51:58.473409 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 14:51:58.473441 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:51:58.475160 (+ 1719us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:51:58.475615 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:51:58.475618 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:51:58.475620 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:51:58.475700 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:51:58.475992 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:51:58.475995 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:51:58.475997 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:51:58.476184 (+ 187us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:51:58.476276 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0927 14:51:58.477239 (+ 963us) server_negotiation.cc:299] Negotiation successful
0927 14:51:58.477365 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48,"spinlock_wait_cycles":896}
I20260927 14:51:58.481592 7768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:51:58.453431 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44660 (local address 127.0.98.60:38741)
0927 14:51:58.453534 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0927 14:51:58.453540 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 14:51:58.473469 (+ 19929us) server_negotiation.cc:407] Connection header received
0927 14:51:58.473629 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:51:58.473634 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:51:58.473698 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:51:58.473749 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:51:58.474711 (+ 962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:51:58.475500 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:51:58.476609 (+ 1109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:51:58.476944 (+ 335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:51:58.477019 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:51:58.478601 (+ 1582us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:51:58.478604 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:51:58.478607 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 14:51:58.478631 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:51:58.480019 (+ 1388us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:51:58.480412 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:51:58.480417 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:51:58.480419 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:51:58.480498 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:51:58.480764 (+ 266us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:51:58.480767 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:51:58.480769 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:51:58.480931 (+ 162us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:51:58.481055 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0927 14:51:58.481300 (+ 245us) server_negotiation.cc:299] Negotiation successful
0927 14:51:58.481424 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28,"spinlock_wait_cycles":2432}
I20260927 14:51:59.437726 7819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:51:59.437321 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36527 (local address 127.0.98.3:59577)
0927 14:51:59.437557 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0927 14:51:59.437633 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36527: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":166,"thread_start_us":70,"threads_started":1}
W20260927 14:51:59.438115 7755 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:36527 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.98.62:36527: connect: Connection refused (error 111)
I20260927 14:51:59.439174 7820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:51:59.438774 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36527 (local address 127.0.98.2:58919)
0927 14:51:59.438984 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0927 14:51:59.439079 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36527: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":83,"threads_started":1}
W20260927 14:51:59.439445 7618 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:36527 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.98.62:36527: connect: Connection refused (error 111)
I20260927 14:51:59.440375 7821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:51:59.439951 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36527 (local address 127.0.98.1:56895)
0927 14:51:59.440191 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0927 14:51:59.440276 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36527: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":166,"thread_start_us":70,"threads_started":1}
W20260927 14:51:59.440722 7481 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:36527 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.98.62:36527: connect: Connection refused (error 111)
I20260927 14:51:59.951514 7822 raft_consensus.cc:482] T 00000000000000000000000000000000 P 34720d7704db415f8c02e5d750103c4a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d60b79f337e343bca73eadfa1bd55f6a)
I20260927 14:51:59.951782 7822 raft_consensus.cc:504] T 00000000000000000000000000000000 P 34720d7704db415f8c02e5d750103c4a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "d60b79f337e343bca73eadfa1bd55f6a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36527 } } peers { permanent_uuid: "413c118097b64cd8af3711ec05bf636a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37119 } attrs { promote: false } } peers { permanent_uuid: "34720d7704db415f8c02e5d750103c4a" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38741 } attrs { promote: false } }
I20260927 14:51:59.952824 7822 leader_election.cc:288] T 00000000000000000000000000000000 P 34720d7704db415f8c02e5d750103c4a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d60b79f337e343bca73eadfa1bd55f6a (127.0.98.62:36527), 413c118097b64cd8af3711ec05bf636a (127.0.98.61:37119)
I20260927 14:51:59.953034 7823 raft_consensus.cc:482] T 00000000000000000000000000000000 P 413c118097b64cd8af3711ec05bf636a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d60b79f337e343bca73eadfa1bd55f6a)
I20260927 14:51:59.953269 7823 raft_consensus.cc:504] T 00000000000000000000000000000000 P 413c118097b64cd8af3711ec05bf636a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "d60b79f337e343bca73eadfa1bd55f6a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36527 } } peers { permanent_uuid: "413c118097b64cd8af3711ec05bf636a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37119 } attrs { promote: false } } peers { permanent_uuid: "34720d7704db415f8c02e5d750103c4a" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38741 } attrs { promote: false } }
I20260927 14:51:59.953576 7825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:51:59.953013 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36527 (local address 127.0.0.1:40810)
0927 14:51:59.953395 (+ 382us) negotiation.cc:107] Waiting for socket to connect
0927 14:51:59.953480 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36527: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":286,"thread_start_us":71,"threads_started":1}
W20260927 14:51:59.953735 7286 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36527: connect: Connection refused (error 111)
I20260927 14:51:59.954385 7823 leader_election.cc:288] T 00000000000000000000000000000000 P 413c118097b64cd8af3711ec05bf636a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d60b79f337e343bca73eadfa1bd55f6a (127.0.98.62:36527), 34720d7704db415f8c02e5d750103c4a (127.0.98.60:38741)
I20260927 14:51:59.954591 7825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:51:59.954326 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36527 (local address 127.0.0.1:40820)
0927 14:51:59.954465 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0927 14:51:59.954518 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36527: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
I20260927 14:51:59.954880 7828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:51:59.954476 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36527 (local address 127.0.0.1:40824)
0927 14:51:59.954730 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0927 14:51:59.954796 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36527: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":64,"threads_started":1}
W20260927 14:51:59.954887 7286 leader_election.cc:334] T 00000000000000000000000000000000 P 34720d7704db415f8c02e5d750103c4a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer d60b79f337e343bca73eadfa1bd55f6a (127.0.98.62:36527): Network error: Client connection negotiation failed: client connection to 127.0.98.62:36527: connect: Connection refused (error 111)
W20260927 14:51:59.955044 7104 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36527: connect: Connection refused (error 111)
I20260927 14:51:59.955883 7828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:51:59.955660 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36527 (local address 127.0.0.1:40834)
0927 14:51:59.955772 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0927 14:51:59.955822 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36527: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260927 14:51:59.955993 7104 leader_election.cc:334] T 00000000000000000000000000000000 P 413c118097b64cd8af3711ec05bf636a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer d60b79f337e343bca73eadfa1bd55f6a (127.0.98.62:36527): Network error: Client connection negotiation failed: client connection to 127.0.98.62:36527: connect: Connection refused (error 111)
I20260927 14:51:59.957907 7824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:51:59.953115 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:37119 (local address 127.0.0.1:52242)
0927 14:51:59.953433 (+ 318us) negotiation.cc:107] Waiting for socket to connect
0927 14:51:59.953453 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0927 14:51:59.953586 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:51:59.954022 (+ 436us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:51:59.954025 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:51:59.954045 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 14:51:59.954146 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:51:59.954153 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:51:59.955457 (+ 1304us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:51:59.955460 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:51:59.956692 (+ 1232us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:51:59.956728 (+ 36us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:51:59.957505 (+ 777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:51:59.957509 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:51:59.957597 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:51:59.957635 (+ 38us) client_negotiation.cc:769] Sending connection context
0927 14:51:59.957716 (+ 81us) client_negotiation.cc:240] Negotiation successful
0927 14:51:59.957781 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":244,"thread_start_us":59,"threads_started":1}
I20260927 14:51:59.957978 7826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:51:59.953115 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52242 (local address 127.0.98.61:37119)
0927 14:51:59.953395 (+ 280us) server_negotiation.cc:206] Beginning negotiation
0927 14:51:59.953400 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 14:51:59.953525 (+ 125us) server_negotiation.cc:407] Connection header received
0927 14:51:59.953731 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:51:59.953734 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:51:59.953786 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:51:59.953881 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 14:51:59.954249 (+ 368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:51:59.955311 (+ 1062us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:51:59.956854 (+ 1543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:51:59.957374 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:51:59.957495 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:51:59.957544 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0927 14:51:59.957765 (+ 221us) server_negotiation.cc:299] Negotiation successful
0927 14:51:59.957854 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":197,"thread_start_us":52,"threads_started":1}
I20260927 14:51:59.958572 7129 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "34720d7704db415f8c02e5d750103c4a" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "413c118097b64cd8af3711ec05bf636a" is_pre_election: true
I20260927 14:51:59.958781 7129 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 413c118097b64cd8af3711ec05bf636a [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 34720d7704db415f8c02e5d750103c4a in term 1.
I20260927 14:51:59.959022 7829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:51:59.954511 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:38741 (local address 127.0.0.1:44684)
0927 14:51:59.954856 (+ 345us) negotiation.cc:107] Waiting for socket to connect
0927 14:51:59.954871 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 14:51:59.955054 (+ 183us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:51:59.955444 (+ 390us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:51:59.955447 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:51:59.955466 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 14:51:59.955539 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:51:59.955544 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:51:59.956419 (+ 875us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:51:59.956423 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:51:59.957569 (+ 1146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:51:59.957580 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:51:59.958620 (+ 1040us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:51:59.958623 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:51:59.958714 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:51:59.958737 (+ 23us) client_negotiation.cc:769] Sending connection context
0927 14:51:59.958820 (+ 83us) client_negotiation.cc:240] Negotiation successful
0927 14:51:59.958893 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":273,"thread_start_us":68,"threads_started":1}
I20260927 14:51:59.959056 7830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:51:59.954731 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44684 (local address 127.0.98.60:38741)
0927 14:51:59.955146 (+ 415us) server_negotiation.cc:206] Beginning negotiation
0927 14:51:59.955158 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 14:51:59.955174 (+ 16us) server_negotiation.cc:407] Connection header received
0927 14:51:59.955214 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:51:59.955216 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:51:59.955290 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:51:59.955370 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 14:51:59.955647 (+ 277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:51:59.956315 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:51:59.957741 (+ 1426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:51:59.958504 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:51:59.958598 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:51:59.958640 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0927 14:51:59.958837 (+ 197us) server_negotiation.cc:299] Negotiation successful
0927 14:51:59.958921 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":349,"thread_start_us":92,"threads_started":1}
I20260927 14:51:59.959209 7287 leader_election.cc:302] T 00000000000000000000000000000000 P 34720d7704db415f8c02e5d750103c4a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 34720d7704db415f8c02e5d750103c4a, 413c118097b64cd8af3711ec05bf636a; no voters: d60b79f337e343bca73eadfa1bd55f6a
I20260927 14:51:59.959451 7822 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 34720d7704db415f8c02e5d750103c4a [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260927 14:51:59.959476 7306 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "413c118097b64cd8af3711ec05bf636a" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "34720d7704db415f8c02e5d750103c4a" is_pre_election: true
I20260927 14:51:59.959582 7822 raft_consensus.cc:482] T 00000000000000000000000000000000 P 34720d7704db415f8c02e5d750103c4a [term 1 FOLLOWER]: Starting leader election (detected failure of leader d60b79f337e343bca73eadfa1bd55f6a)
I20260927 14:51:59.959641 7822 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 34720d7704db415f8c02e5d750103c4a [term 1 FOLLOWER]: Advancing to term 2
I20260927 14:51:59.960618 7822 raft_consensus.cc:504] T 00000000000000000000000000000000 P 34720d7704db415f8c02e5d750103c4a [term 2 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "d60b79f337e343bca73eadfa1bd55f6a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36527 } } peers { permanent_uuid: "413c118097b64cd8af3711ec05bf636a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37119 } attrs { promote: false } } peers { permanent_uuid: "34720d7704db415f8c02e5d750103c4a" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38741 } attrs { promote: false } }
I20260927 14:51:59.960902 7306 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 34720d7704db415f8c02e5d750103c4a [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 413c118097b64cd8af3711ec05bf636a in current term 2: Already voted for candidate 34720d7704db415f8c02e5d750103c4a in this term.
I20260927 14:51:59.961026 7822 leader_election.cc:288] T 00000000000000000000000000000000 P 34720d7704db415f8c02e5d750103c4a [CANDIDATE]: Term 2 election: Requested vote from peers d60b79f337e343bca73eadfa1bd55f6a (127.0.98.62:36527), 413c118097b64cd8af3711ec05bf636a (127.0.98.61:37119)
I20260927 14:51:59.961333 7824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:51:59.961128 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36527 (local address 127.0.0.1:40838)
0927 14:51:59.961209 (+ 81us) negotiation.cc:107] Waiting for socket to connect
0927 14:51:59.961286 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36527: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":22}
I20260927 14:51:59.961449 7129 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "34720d7704db415f8c02e5d750103c4a" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "413c118097b64cd8af3711ec05bf636a"
I20260927 14:51:59.961582 7129 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 413c118097b64cd8af3711ec05bf636a [term 1 FOLLOWER]: Advancing to term 2
I20260927 14:51:59.962455 7129 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 413c118097b64cd8af3711ec05bf636a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 34720d7704db415f8c02e5d750103c4a in term 2.
I20260927 14:51:59.962765 7287 leader_election.cc:302] T 00000000000000000000000000000000 P 34720d7704db415f8c02e5d750103c4a [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 34720d7704db415f8c02e5d750103c4a, 413c118097b64cd8af3711ec05bf636a; no voters:
I20260927 14:51:59.962967 7822 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 34720d7704db415f8c02e5d750103c4a [term 2 FOLLOWER]: Leader election won for term 2
I20260927 14:51:59.962968 7103 leader_election.cc:302] T 00000000000000000000000000000000 P 413c118097b64cd8af3711ec05bf636a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 413c118097b64cd8af3711ec05bf636a; no voters: 34720d7704db415f8c02e5d750103c4a, d60b79f337e343bca73eadfa1bd55f6a
I20260927 14:51:59.963158 7822 raft_consensus.cc:686] T 00000000000000000000000000000000 P 34720d7704db415f8c02e5d750103c4a [term 2 LEADER]: Becoming Leader. State: Replica: 34720d7704db415f8c02e5d750103c4a, State: Running, Role: LEADER
I20260927 14:51:59.963181 7823 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 413c118097b64cd8af3711ec05bf636a [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 14:51:59.963171 7824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:51:59.962961 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36527 (local address 127.0.0.1:40854)
0927 14:51:59.963040 (+ 79us) negotiation.cc:107] Waiting for socket to connect
0927 14:51:59.963083 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36527: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":22}
W20260927 14:51:59.963419 7286 leader_election.cc:334] T 00000000000000000000000000000000 P 34720d7704db415f8c02e5d750103c4a [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer d60b79f337e343bca73eadfa1bd55f6a (127.0.98.62:36527): Network error: Client connection negotiation failed: client connection to 127.0.98.62:36527: connect: Connection refused (error 111)
I20260927 14:51:59.963449 7822 consensus_queue.cc:237] T 00000000000000000000000000000000 P 34720d7704db415f8c02e5d750103c4a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "d60b79f337e343bca73eadfa1bd55f6a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36527 } } peers { permanent_uuid: "413c118097b64cd8af3711ec05bf636a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37119 } attrs { promote: false } } peers { permanent_uuid: "34720d7704db415f8c02e5d750103c4a" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38741 } attrs { promote: false } }
I20260927 14:51:59.964490 7832 sys_catalog.cc:455] T 00000000000000000000000000000000 P 34720d7704db415f8c02e5d750103c4a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 34720d7704db415f8c02e5d750103c4a. Latest consensus state: current_term: 2 leader_uuid: "34720d7704db415f8c02e5d750103c4a" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "d60b79f337e343bca73eadfa1bd55f6a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36527 } } peers { permanent_uuid: "413c118097b64cd8af3711ec05bf636a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37119 } attrs { promote: false } } peers { permanent_uuid: "34720d7704db415f8c02e5d750103c4a" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38741 } attrs { promote: false } } }
I20260927 14:51:59.964615 7832 sys_catalog.cc:458] T 00000000000000000000000000000000 P 34720d7704db415f8c02e5d750103c4a [sys.catalog]: This master's current role is: LEADER
I20260927 14:51:59.964936 7834 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 14:51:59.967149 7834 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 14:51:59.967542 7834 catalog_manager.cc:1259] Loaded cluster ID: 3ea1e4dfc1684ee8ba96f40d83c0eb15
I20260927 14:51:59.967623 7834 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 14:51:59.968155 7834 catalog_manager.cc:1504] Loading token signing keys...
I20260927 14:51:59.968541 7834 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 34720d7704db415f8c02e5d750103c4a: Loaded TSK: 0
I20260927 14:51:59.969266 7834 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 14:51:59.989429 7301 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44674:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 14:51:59.992484 7301 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 14:51:59.995553 7824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:51:59.995306 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36527 (local address 127.0.0.1:40872)
0927 14:51:59.995417 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0927 14:51:59.995486 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36527: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260927 14:51:59.995590 7129 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 413c118097b64cd8af3711ec05bf636a [term 2 FOLLOWER]: Refusing update from remote peer 34720d7704db415f8c02e5d750103c4a: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 2 index: 14. (index mismatch)
I20260927 14:51:59.996143 7832 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 34720d7704db415f8c02e5d750103c4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "413c118097b64cd8af3711ec05bf636a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37119 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260927 14:51:59.996301 7824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:51:59.996063 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36527 (local address 127.0.0.1:40880)
0927 14:51:59.996163 (+ 100us) negotiation.cc:107] Waiting for socket to connect
0927 14:51:59.996209 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36527: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260927 14:51:59.996400 7286 consensus_peers.cc:487] T 00000000000000000000000000000000 P 34720d7704db415f8c02e5d750103c4a -> Peer d60b79f337e343bca73eadfa1bd55f6a (127.0.98.62:36527): Couldn't send request to peer d60b79f337e343bca73eadfa1bd55f6a. Status: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36527: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260927 14:51:59.997726 7823 sys_catalog.cc:455] T 00000000000000000000000000000000 P 413c118097b64cd8af3711ec05bf636a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 34720d7704db415f8c02e5d750103c4a. Latest consensus state: current_term: 2 leader_uuid: "34720d7704db415f8c02e5d750103c4a" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "d60b79f337e343bca73eadfa1bd55f6a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36527 } } peers { permanent_uuid: "413c118097b64cd8af3711ec05bf636a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37119 } attrs { promote: false } } peers { permanent_uuid: "34720d7704db415f8c02e5d750103c4a" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38741 } attrs { promote: false } } }
I20260927 14:51:59.997888 7823 sys_catalog.cc:458] T 00000000000000000000000000000000 P 413c118097b64cd8af3711ec05bf636a [sys.catalog]: This master's current role is: FOLLOWER
I20260927 14:51:59.998965 7832 sys_catalog.cc:455] T 00000000000000000000000000000000 P 34720d7704db415f8c02e5d750103c4a [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "34720d7704db415f8c02e5d750103c4a" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "d60b79f337e343bca73eadfa1bd55f6a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36527 } } peers { permanent_uuid: "413c118097b64cd8af3711ec05bf636a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37119 } attrs { promote: false } } peers { permanent_uuid: "34720d7704db415f8c02e5d750103c4a" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38741 } attrs { promote: false } } }
I20260927 14:51:59.999089 7832 sys_catalog.cc:458] T 00000000000000000000000000000000 P 34720d7704db415f8c02e5d750103c4a [sys.catalog]: This master's current role is: LEADER
I20260927 14:51:59.999471 7823 sys_catalog.cc:455] T 00000000000000000000000000000000 P 413c118097b64cd8af3711ec05bf636a [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "34720d7704db415f8c02e5d750103c4a" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "d60b79f337e343bca73eadfa1bd55f6a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36527 } } peers { permanent_uuid: "413c118097b64cd8af3711ec05bf636a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37119 } attrs { promote: false } } peers { permanent_uuid: "34720d7704db415f8c02e5d750103c4a" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38741 } attrs { promote: false } } }
I20260927 14:51:59.999557 7836 mvcc.cc:205] Tried to move back new op lower bound from 7333972869097009152 to 7333972868972986368. Current Snapshot: MvccSnapshot[applied={T|T < 7333972869097009152}]
I20260927 14:51:59.999594 7823 sys_catalog.cc:458] T 00000000000000000000000000000000 P 413c118097b64cd8af3711ec05bf636a [sys.catalog]: This master's current role is: FOLLOWER
I20260927 14:52:00.009940 7842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:52:00.005645 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33956 (local address 127.0.98.1:43017)
0927 14:52:00.005915 (+ 270us) server_negotiation.cc:206] Beginning negotiation
0927 14:52:00.005920 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 14:52:00.005936 (+ 16us) server_negotiation.cc:407] Connection header received
0927 14:52:00.005988 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:52:00.005991 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:52:00.006038 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:52:00.006132 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 14:52:00.006365 (+ 233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:52:00.007110 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:52:00.008719 (+ 1609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:52:00.009354 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:52:00.009489 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:52:00.009554 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0927 14:52:00.009727 (+ 173us) server_negotiation.cc:299] Negotiation successful
0927 14:52:00.009797 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":196,"thread_start_us":97,"threads_started":1}
I20260927 14:52:00.009939 7825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:52:00.005400 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:43017 (local address 127.0.0.1:33956)
0927 14:52:00.005642 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0927 14:52:00.005657 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 14:52:00.005739 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:52:00.006145 (+ 406us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:52:00.006148 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:52:00.006166 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 14:52:00.006253 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:52:00.006258 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:52:00.007233 (+ 975us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:52:00.007236 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:52:00.008590 (+ 1354us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:52:00.008599 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:52:00.009489 (+ 890us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:52:00.009492 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:52:00.009583 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:52:00.009605 (+ 22us) client_negotiation.cc:769] Sending connection context
0927 14:52:00.009702 (+ 97us) client_negotiation.cc:240] Negotiation successful
0927 14:52:00.009776 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":177,"mutex_wait_us":86}
I20260927 14:52:00.010210 7844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:52:00.006029 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32888 (local address 127.0.98.3:38555)
0927 14:52:00.006568 (+ 539us) server_negotiation.cc:206] Beginning negotiation
0927 14:52:00.006573 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 14:52:00.006787 (+ 214us) server_negotiation.cc:407] Connection header received
0927 14:52:00.006886 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:52:00.006889 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:52:00.006936 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:52:00.007018 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 14:52:00.007269 (+ 251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:52:00.007758 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:52:00.008753 (+ 995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:52:00.009410 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:52:00.009533 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:52:00.009610 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0927 14:52:00.009988 (+ 378us) server_negotiation.cc:299] Negotiation successful
0927 14:52:00.010079 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":463,"thread_start_us":71,"threads_started":1}
I20260927 14:52:00.010249 7824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:52:00.005127 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:34621 (local address 127.0.0.1:40214)
0927 14:52:00.005269 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0927 14:52:00.005284 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 14:52:00.005366 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:52:00.005973 (+ 607us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:52:00.005977 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:52:00.005993 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 14:52:00.006128 (+ 135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:52:00.006137 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:52:00.007170 (+ 1033us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:52:00.007175 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:52:00.008590 (+ 1415us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:52:00.008599 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:52:00.009599 (+ 1000us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:52:00.009603 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:52:00.009691 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:52:00.009714 (+ 23us) client_negotiation.cc:769] Sending connection context
0927 14:52:00.009760 (+ 46us) client_negotiation.cc:240] Negotiation successful
0927 14:52:00.009844 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":78,"mutex_wait_us":110}
I20260927 14:52:00.010447 7843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:52:00.005697 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:38555 (local address 127.0.0.1:32888)
0927 14:52:00.006688 (+ 991us) negotiation.cc:107] Waiting for socket to connect
0927 14:52:00.006701 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 14:52:00.006791 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:52:00.007039 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:52:00.007042 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:52:00.007056 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 14:52:00.007136 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:52:00.007145 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:52:00.007888 (+ 743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:52:00.007893 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:52:00.008625 (+ 732us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:52:00.008633 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:52:00.009656 (+ 1023us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:52:00.009659 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:52:00.009732 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:52:00.009761 (+ 29us) client_negotiation.cc:769] Sending connection context
0927 14:52:00.009818 (+ 57us) client_negotiation.cc:240] Negotiation successful
0927 14:52:00.009870 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":939,"thread_start_us":71,"threads_started":1}
I20260927 14:52:00.010497 7414 tablet_service.cc:1467] Processing CreateTablet for tablet 56f8151bb1b14a95a2d93b144e458e22 (DEFAULT_TABLE table=test-table [id=75b74236ec1646ecb7bbbfc4079a9da6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 14:52:00.010767 7841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:52:00.005238 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40214 (local address 127.0.98.2:34621)
0927 14:52:00.005712 (+ 474us) server_negotiation.cc:206] Beginning negotiation
0927 14:52:00.005717 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 14:52:00.005730 (+ 13us) server_negotiation.cc:407] Connection header received
0927 14:52:00.005783 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:52:00.005789 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:52:00.005838 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:52:00.006240 (+ 402us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 14:52:00.006363 (+ 123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:52:00.007082 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:52:00.008847 (+ 1765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:52:00.009476 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:52:00.009965 (+ 489us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:52:00.010538 (+ 573us) server_negotiation.cc:1035] Waiting for connection context
0927 14:52:00.010579 (+ 41us) server_negotiation.cc:299] Negotiation successful
0927 14:52:00.010655 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":389,"thread_start_us":89,"threads_started":1}
I20260927 14:52:00.010792 7414 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 56f8151bb1b14a95a2d93b144e458e22. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 14:52:00.011270 7551 tablet_service.cc:1467] Processing CreateTablet for tablet 56f8151bb1b14a95a2d93b144e458e22 (DEFAULT_TABLE table=test-table [id=75b74236ec1646ecb7bbbfc4079a9da6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 14:52:00.011518 7551 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 56f8151bb1b14a95a2d93b144e458e22. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 14:52:00.013417 7845 tablet_bootstrap.cc:492] T 56f8151bb1b14a95a2d93b144e458e22 P f37a65090f8b4b8f9a4facdb930c9eec: Bootstrap starting.
I20260927 14:52:00.013652 7688 tablet_service.cc:1467] Processing CreateTablet for tablet 56f8151bb1b14a95a2d93b144e458e22 (DEFAULT_TABLE table=test-table [id=75b74236ec1646ecb7bbbfc4079a9da6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 14:52:00.013921 7688 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 56f8151bb1b14a95a2d93b144e458e22. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 14:52:00.014122 7846 tablet_bootstrap.cc:492] T 56f8151bb1b14a95a2d93b144e458e22 P a473db7856d24a8ba3efb9bbc9e12de9: Bootstrap starting.
I20260927 14:52:00.014420 7845 tablet_bootstrap.cc:654] T 56f8151bb1b14a95a2d93b144e458e22 P f37a65090f8b4b8f9a4facdb930c9eec: Neither blocks nor log segments found. Creating new log.
I20260927 14:52:00.015058 7846 tablet_bootstrap.cc:654] T 56f8151bb1b14a95a2d93b144e458e22 P a473db7856d24a8ba3efb9bbc9e12de9: Neither blocks nor log segments found. Creating new log.
I20260927 14:52:00.015661 7845 tablet_bootstrap.cc:492] T 56f8151bb1b14a95a2d93b144e458e22 P f37a65090f8b4b8f9a4facdb930c9eec: No bootstrap required, opened a new log
I20260927 14:52:00.015767 7845 ts_tablet_manager.cc:1397] T 56f8151bb1b14a95a2d93b144e458e22 P f37a65090f8b4b8f9a4facdb930c9eec: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260927 14:52:00.016266 7845 raft_consensus.cc:348] T 56f8151bb1b14a95a2d93b144e458e22 P f37a65090f8b4b8f9a4facdb930c9eec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a473db7856d24a8ba3efb9bbc9e12de9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34621 } } peers { permanent_uuid: "f37a65090f8b4b8f9a4facdb930c9eec" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43017 } } peers { permanent_uuid: "df8799b0da0c4b4e895000a21761211e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38555 } }
I20260927 14:52:00.016394 7846 tablet_bootstrap.cc:492] T 56f8151bb1b14a95a2d93b144e458e22 P a473db7856d24a8ba3efb9bbc9e12de9: No bootstrap required, opened a new log
I20260927 14:52:00.016390 7845 raft_consensus.cc:374] T 56f8151bb1b14a95a2d93b144e458e22 P f37a65090f8b4b8f9a4facdb930c9eec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 14:52:00.016441 7845 raft_consensus.cc:729] T 56f8151bb1b14a95a2d93b144e458e22 P f37a65090f8b4b8f9a4facdb930c9eec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f37a65090f8b4b8f9a4facdb930c9eec, State: Initialized, Role: FOLLOWER
I20260927 14:52:00.016485 7846 ts_tablet_manager.cc:1397] T 56f8151bb1b14a95a2d93b144e458e22 P a473db7856d24a8ba3efb9bbc9e12de9: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260927 14:52:00.016597 7845 consensus_queue.cc:260] T 56f8151bb1b14a95a2d93b144e458e22 P f37a65090f8b4b8f9a4facdb930c9eec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a473db7856d24a8ba3efb9bbc9e12de9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34621 } } peers { permanent_uuid: "f37a65090f8b4b8f9a4facdb930c9eec" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43017 } } peers { permanent_uuid: "df8799b0da0c4b4e895000a21761211e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38555 } }
I20260927 14:52:00.016873 7846 raft_consensus.cc:348] T 56f8151bb1b14a95a2d93b144e458e22 P a473db7856d24a8ba3efb9bbc9e12de9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a473db7856d24a8ba3efb9bbc9e12de9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34621 } } peers { permanent_uuid: "f37a65090f8b4b8f9a4facdb930c9eec" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43017 } } peers { permanent_uuid: "df8799b0da0c4b4e895000a21761211e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38555 } }
I20260927 14:52:00.016948 7845 ts_tablet_manager.cc:1428] T 56f8151bb1b14a95a2d93b144e458e22 P f37a65090f8b4b8f9a4facdb930c9eec: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20260927 14:52:00.016983 7846 raft_consensus.cc:374] T 56f8151bb1b14a95a2d93b144e458e22 P a473db7856d24a8ba3efb9bbc9e12de9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 14:52:00.017014 7846 raft_consensus.cc:729] T 56f8151bb1b14a95a2d93b144e458e22 P a473db7856d24a8ba3efb9bbc9e12de9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a473db7856d24a8ba3efb9bbc9e12de9, State: Initialized, Role: FOLLOWER
I20260927 14:52:00.017127 7849 tablet_bootstrap.cc:492] T 56f8151bb1b14a95a2d93b144e458e22 P df8799b0da0c4b4e895000a21761211e: Bootstrap starting.
I20260927 14:52:00.017203 7846 consensus_queue.cc:260] T 56f8151bb1b14a95a2d93b144e458e22 P a473db7856d24a8ba3efb9bbc9e12de9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a473db7856d24a8ba3efb9bbc9e12de9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34621 } } peers { permanent_uuid: "f37a65090f8b4b8f9a4facdb930c9eec" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43017 } } peers { permanent_uuid: "df8799b0da0c4b4e895000a21761211e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38555 } }
I20260927 14:52:00.017596 7846 ts_tablet_manager.cc:1428] T 56f8151bb1b14a95a2d93b144e458e22 P a473db7856d24a8ba3efb9bbc9e12de9: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260927 14:52:00.018000 7849 tablet_bootstrap.cc:654] T 56f8151bb1b14a95a2d93b144e458e22 P df8799b0da0c4b4e895000a21761211e: Neither blocks nor log segments found. Creating new log.
I20260927 14:52:00.018983 7852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:52:00.017379 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36527 (local address 127.0.98.1:39975)
0927 14:52:00.018835 (+ 1456us) negotiation.cc:107] Waiting for socket to connect
0927 14:52:00.018899 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36527: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1377,"thread_start_us":86,"threads_started":1}
I20260927 14:52:00.019423 7480 heartbeater.cc:503] Master 127.0.98.60:38741 was elected leader, sending a full tablet report...
I20260927 14:52:00.019645 7854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:52:00.018717 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36527 (local address 127.0.98.2:35493)
0927 14:52:00.019492 (+ 775us) negotiation.cc:107] Waiting for socket to connect
0927 14:52:00.019546 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36527: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":689,"thread_start_us":82,"threads_started":1}
I20260927 14:52:00.019888 7617 heartbeater.cc:503] Master 127.0.98.60:38741 was elected leader, sending a full tablet report...
I20260927 14:52:00.020396 7854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:52:00.020026 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36527 (local address 127.0.98.2:46689)
0927 14:52:00.020277 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0927 14:52:00.020325 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36527: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":73,"threads_started":1}
I20260927 14:52:00.020457 7849 tablet_bootstrap.cc:492] T 56f8151bb1b14a95a2d93b144e458e22 P df8799b0da0c4b4e895000a21761211e: No bootstrap required, opened a new log
I20260927 14:52:00.020531 7849 ts_tablet_manager.cc:1397] T 56f8151bb1b14a95a2d93b144e458e22 P df8799b0da0c4b4e895000a21761211e: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260927 14:52:00.020884 7849 raft_consensus.cc:348] T 56f8151bb1b14a95a2d93b144e458e22 P df8799b0da0c4b4e895000a21761211e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a473db7856d24a8ba3efb9bbc9e12de9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34621 } } peers { permanent_uuid: "f37a65090f8b4b8f9a4facdb930c9eec" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43017 } } peers { permanent_uuid: "df8799b0da0c4b4e895000a21761211e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38555 } }
I20260927 14:52:00.020983 7849 raft_consensus.cc:374] T 56f8151bb1b14a95a2d93b144e458e22 P df8799b0da0c4b4e895000a21761211e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 14:52:00.021023 7849 raft_consensus.cc:729] T 56f8151bb1b14a95a2d93b144e458e22 P df8799b0da0c4b4e895000a21761211e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df8799b0da0c4b4e895000a21761211e, State: Initialized, Role: FOLLOWER
I20260927 14:52:00.021245 7849 consensus_queue.cc:260] T 56f8151bb1b14a95a2d93b144e458e22 P df8799b0da0c4b4e895000a21761211e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a473db7856d24a8ba3efb9bbc9e12de9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34621 } } peers { permanent_uuid: "f37a65090f8b4b8f9a4facdb930c9eec" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43017 } } peers { permanent_uuid: "df8799b0da0c4b4e895000a21761211e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38555 } }
I20260927 14:52:00.021550 7849 ts_tablet_manager.cc:1428] T 56f8151bb1b14a95a2d93b144e458e22 P df8799b0da0c4b4e895000a21761211e: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260927 14:52:00.022634 7857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:52:00.022257 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36527 (local address 127.0.98.3:39521)
0927 14:52:00.022502 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0927 14:52:00.022553 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36527: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":91,"threads_started":1}
I20260927 14:52:00.023314 7857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:52:00.023118 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36527 (local address 127.0.98.3:52411)
0927 14:52:00.023215 (+ 97us) negotiation.cc:107] Waiting for socket to connect
0927 14:52:00.023261 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36527: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260927 14:52:00.024017 7620 tablet.cc:2369] T 56f8151bb1b14a95a2d93b144e458e22 P a473db7856d24a8ba3efb9bbc9e12de9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 14:52:00.026149 7852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:52:00.025898 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36527 (local address 127.0.98.1:49889)
0927 14:52:00.026034 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0927 14:52:00.026083 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36527: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
I20260927 14:52:00.041270 7851 raft_consensus.cc:482] T 56f8151bb1b14a95a2d93b144e458e22 P a473db7856d24a8ba3efb9bbc9e12de9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 14:52:00.041404 7851 raft_consensus.cc:504] T 56f8151bb1b14a95a2d93b144e458e22 P a473db7856d24a8ba3efb9bbc9e12de9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a473db7856d24a8ba3efb9bbc9e12de9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34621 } } peers { permanent_uuid: "f37a65090f8b4b8f9a4facdb930c9eec" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43017 } } peers { permanent_uuid: "df8799b0da0c4b4e895000a21761211e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38555 } }
I20260927 14:52:00.042586 7851 leader_election.cc:288] T 56f8151bb1b14a95a2d93b144e458e22 P a473db7856d24a8ba3efb9bbc9e12de9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f37a65090f8b4b8f9a4facdb930c9eec (127.0.98.1:43017), df8799b0da0c4b4e895000a21761211e (127.0.98.3:38555)
I20260927 14:52:00.046142 7855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:52:00.042732 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:43017 (local address 127.0.98.2:54277)
0927 14:52:00.042899 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0927 14:52:00.042916 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 14:52:00.043032 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:52:00.043294 (+ 262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:52:00.043297 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:52:00.043312 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 14:52:00.043465 (+ 153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:52:00.043471 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:52:00.044181 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:52:00.044188 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:52:00.045010 (+ 822us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:52:00.045018 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:52:00.045694 (+ 676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:52:00.045698 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:52:00.045798 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:52:00.045819 (+ 21us) client_negotiation.cc:769] Sending connection context
0927 14:52:00.045887 (+ 68us) client_negotiation.cc:240] Negotiation successful
0927 14:52:00.045969 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":90,"mutex_wait_us":29}
I20260927 14:52:00.046182 7844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:52:00.042927 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:38343 (local address 127.0.98.3:38555)
0927 14:52:00.043049 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0927 14:52:00.043055 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 14:52:00.043069 (+ 14us) server_negotiation.cc:407] Connection header received
0927 14:52:00.043188 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:52:00.043197 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:52:00.043238 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:52:00.043313 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 14:52:00.043616 (+ 303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:52:00.044114 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:52:00.045199 (+ 1085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:52:00.045590 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:52:00.045711 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:52:00.045774 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0927 14:52:00.045998 (+ 224us) server_negotiation.cc:299] Negotiation successful
0927 14:52:00.046081 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260927 14:52:00.046140 7842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:52:00.042854 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:54277 (local address 127.0.98.1:43017)
0927 14:52:00.042971 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0927 14:52:00.042977 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 14:52:00.042993 (+ 16us) server_negotiation.cc:407] Connection header received
0927 14:52:00.043151 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:52:00.043159 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:52:00.043201 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:52:00.043284 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 14:52:00.043586 (+ 302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:52:00.044075 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:52:00.045205 (+ 1130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:52:00.045574 (+ 369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:52:00.045686 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:52:00.045753 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0927 14:52:00.045932 (+ 179us) server_negotiation.cc:299] Negotiation successful
0927 14:52:00.046014 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260927 14:52:00.046144 7854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:52:00.042825 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:38555 (local address 127.0.98.2:38343)
0927 14:52:00.042983 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0927 14:52:00.042997 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 14:52:00.043083 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:52:00.043333 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:52:00.043336 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:52:00.043350 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 14:52:00.043510 (+ 160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:52:00.043516 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:52:00.044230 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:52:00.044233 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:52:00.045016 (+ 783us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:52:00.045025 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:52:00.045747 (+ 722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:52:00.045751 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:52:00.045852 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:52:00.045890 (+ 38us) client_negotiation.cc:769] Sending connection context
0927 14:52:00.045968 (+ 78us) client_negotiation.cc:240] Negotiation successful
0927 14:52:00.046025 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":90,"mutex_wait_us":44}
I20260927 14:52:00.046608 7434 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56f8151bb1b14a95a2d93b144e458e22" candidate_uuid: "a473db7856d24a8ba3efb9bbc9e12de9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f37a65090f8b4b8f9a4facdb930c9eec" is_pre_election: true
I20260927 14:52:00.046681 7708 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56f8151bb1b14a95a2d93b144e458e22" candidate_uuid: "a473db7856d24a8ba3efb9bbc9e12de9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df8799b0da0c4b4e895000a21761211e" is_pre_election: true
I20260927 14:52:00.046753 7434 raft_consensus.cc:2454] T 56f8151bb1b14a95a2d93b144e458e22 P f37a65090f8b4b8f9a4facdb930c9eec [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a473db7856d24a8ba3efb9bbc9e12de9 in term 0.
I20260927 14:52:00.046944 7708 raft_consensus.cc:2454] T 56f8151bb1b14a95a2d93b144e458e22 P df8799b0da0c4b4e895000a21761211e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a473db7856d24a8ba3efb9bbc9e12de9 in term 0.
I20260927 14:52:00.047190 7506 leader_election.cc:302] T 56f8151bb1b14a95a2d93b144e458e22 P a473db7856d24a8ba3efb9bbc9e12de9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a473db7856d24a8ba3efb9bbc9e12de9, f37a65090f8b4b8f9a4facdb930c9eec; no voters:
I20260927 14:52:00.047482 7851 raft_consensus.cc:2781] T 56f8151bb1b14a95a2d93b144e458e22 P a473db7856d24a8ba3efb9bbc9e12de9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 14:52:00.047551 7851 raft_consensus.cc:482] T 56f8151bb1b14a95a2d93b144e458e22 P a473db7856d24a8ba3efb9bbc9e12de9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 14:52:00.047605 7851 raft_consensus.cc:3037] T 56f8151bb1b14a95a2d93b144e458e22 P a473db7856d24a8ba3efb9bbc9e12de9 [term 0 FOLLOWER]: Advancing to term 1
I20260927 14:52:00.048537 7851 raft_consensus.cc:504] T 56f8151bb1b14a95a2d93b144e458e22 P a473db7856d24a8ba3efb9bbc9e12de9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a473db7856d24a8ba3efb9bbc9e12de9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34621 } } peers { permanent_uuid: "f37a65090f8b4b8f9a4facdb930c9eec" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43017 } } peers { permanent_uuid: "df8799b0da0c4b4e895000a21761211e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38555 } }
I20260927 14:52:00.048857 7851 leader_election.cc:288] T 56f8151bb1b14a95a2d93b144e458e22 P a473db7856d24a8ba3efb9bbc9e12de9 [CANDIDATE]: Term 1 election: Requested vote from peers f37a65090f8b4b8f9a4facdb930c9eec (127.0.98.1:43017), df8799b0da0c4b4e895000a21761211e (127.0.98.3:38555)
I20260927 14:52:00.049230 7434 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56f8151bb1b14a95a2d93b144e458e22" candidate_uuid: "a473db7856d24a8ba3efb9bbc9e12de9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f37a65090f8b4b8f9a4facdb930c9eec"
I20260927 14:52:00.049332 7434 raft_consensus.cc:3037] T 56f8151bb1b14a95a2d93b144e458e22 P f37a65090f8b4b8f9a4facdb930c9eec [term 0 FOLLOWER]: Advancing to term 1
I20260927 14:52:00.050190 7434 raft_consensus.cc:2454] T 56f8151bb1b14a95a2d93b144e458e22 P f37a65090f8b4b8f9a4facdb930c9eec [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a473db7856d24a8ba3efb9bbc9e12de9 in term 1.
I20260927 14:52:00.050448 7708 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56f8151bb1b14a95a2d93b144e458e22" candidate_uuid: "a473db7856d24a8ba3efb9bbc9e12de9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df8799b0da0c4b4e895000a21761211e"
I20260927 14:52:00.050499 7506 leader_election.cc:302] T 56f8151bb1b14a95a2d93b144e458e22 P a473db7856d24a8ba3efb9bbc9e12de9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a473db7856d24a8ba3efb9bbc9e12de9, f37a65090f8b4b8f9a4facdb930c9eec; no voters:
I20260927 14:52:00.050577 7708 raft_consensus.cc:3037] T 56f8151bb1b14a95a2d93b144e458e22 P df8799b0da0c4b4e895000a21761211e [term 0 FOLLOWER]: Advancing to term 1
I20260927 14:52:00.050685 7851 raft_consensus.cc:2781] T 56f8151bb1b14a95a2d93b144e458e22 P a473db7856d24a8ba3efb9bbc9e12de9 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 14:52:00.050920 7851 raft_consensus.cc:686] T 56f8151bb1b14a95a2d93b144e458e22 P a473db7856d24a8ba3efb9bbc9e12de9 [term 1 LEADER]: Becoming Leader. State: Replica: a473db7856d24a8ba3efb9bbc9e12de9, State: Running, Role: LEADER
I20260927 14:52:00.051247 7851 consensus_queue.cc:237] T 56f8151bb1b14a95a2d93b144e458e22 P a473db7856d24a8ba3efb9bbc9e12de9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a473db7856d24a8ba3efb9bbc9e12de9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34621 } } peers { permanent_uuid: "f37a65090f8b4b8f9a4facdb930c9eec" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43017 } } peers { permanent_uuid: "df8799b0da0c4b4e895000a21761211e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38555 } }
I20260927 14:52:00.051776 7708 raft_consensus.cc:2454] T 56f8151bb1b14a95a2d93b144e458e22 P df8799b0da0c4b4e895000a21761211e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a473db7856d24a8ba3efb9bbc9e12de9 in term 1.
I20260927 14:52:00.052121 7855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:52:00.051279 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36527 (local address 127.0.98.2:59789)
0927 14:52:00.051999 (+ 720us) negotiation.cc:107] Waiting for socket to connect
0927 14:52:00.052054 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36527: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":600}
I20260927 14:52:00.053793 7300 catalog_manager.cc:5495] T 56f8151bb1b14a95a2d93b144e458e22 P a473db7856d24a8ba3efb9bbc9e12de9 reported cstate change: term changed from 0 to 1, leader changed from <none> to a473db7856d24a8ba3efb9bbc9e12de9 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "a473db7856d24a8ba3efb9bbc9e12de9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a473db7856d24a8ba3efb9bbc9e12de9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34621 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f37a65090f8b4b8f9a4facdb930c9eec" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43017 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "df8799b0da0c4b4e895000a21761211e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38555 } health_report { overall_health: UNKNOWN } } }
I20260927 14:52:00.074090 7841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:52:00.070452 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40216 (local address 127.0.98.2:34621)
0927 14:52:00.070592 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0927 14:52:00.070595 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 14:52:00.070662 (+ 67us) server_negotiation.cc:407] Connection header received
0927 14:52:00.070806 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:52:00.070808 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:52:00.070884 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:52:00.070982 (+ 98us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 14:52:00.071227 (+ 245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:52:00.071722 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:52:00.072575 (+ 853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:52:00.072794 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:52:00.072891 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:52:00.073217 (+ 326us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 14:52:00.073646 (+ 429us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 14:52:00.073732 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0927 14:52:00.073865 (+ 133us) server_negotiation.cc:299] Negotiation successful
0927 14:52:00.073912 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260927 14:52:00.077028 7434 raft_consensus.cc:1261] T 56f8151bb1b14a95a2d93b144e458e22 P f37a65090f8b4b8f9a4facdb930c9eec [term 1 FOLLOWER]: Refusing update from remote peer a473db7856d24a8ba3efb9bbc9e12de9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 14:52:00.077361 7708 raft_consensus.cc:1261] T 56f8151bb1b14a95a2d93b144e458e22 P df8799b0da0c4b4e895000a21761211e [term 1 FOLLOWER]: Refusing update from remote peer a473db7856d24a8ba3efb9bbc9e12de9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 14:52:00.077616 7858 consensus_queue.cc:1035] T 56f8151bb1b14a95a2d93b144e458e22 P a473db7856d24a8ba3efb9bbc9e12de9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f37a65090f8b4b8f9a4facdb930c9eec" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43017 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 14:52:00.077943 7851 consensus_queue.cc:1035] T 56f8151bb1b14a95a2d93b144e458e22 P a473db7856d24a8ba3efb9bbc9e12de9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df8799b0da0c4b4e895000a21761211e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38555 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 14:52:00.079464 7852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:52:00.078999 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36527 (local address 127.0.98.1:51383)
0927 14:52:00.079314 (+ 315us) negotiation.cc:107] Waiting for socket to connect
0927 14:52:00.079378 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36527: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":247}
I20260927 14:52:00.079732 7754 heartbeater.cc:503] Master 127.0.98.60:38741 was elected leader, sending a full tablet report...
I20260927 14:52:00.080531 7857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:52:00.080238 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36527 (local address 127.0.98.3:38003)
0927 14:52:00.080387 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0927 14:52:00.080439 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36527: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85}
I20260927 14:52:00.083114 7852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:52:00.082854 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36527 (local address 127.0.98.1:41843)
0927 14:52:00.083000 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0927 14:52:00.083050 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36527: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88}
I20260927 14:52:00.084232 7857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:52:00.083983 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36527 (local address 127.0.98.3:45961)
0927 14:52:00.084124 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0927 14:52:00.084169 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36527: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
I20260927 14:52:00.085847 7855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:52:00.082320 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36527 (local address 127.0.98.2:57349)
0927 14:52:00.085716 (+ 3396us) negotiation.cc:107] Waiting for socket to connect
0927 14:52:00.085769 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36527: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":3328}
I20260927 14:52:00.090353 7300 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44674:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 14:52:00.090489 7300 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44674:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 14:52:00.094245 7300 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 34720d7704db415f8c02e5d750103c4a: Sending DeleteTablet for 3 replicas of tablet 56f8151bb1b14a95a2d93b144e458e22
I20260927 14:52:00.094882 7551 tablet_service.cc:1514] Processing DeleteTablet for tablet 56f8151bb1b14a95a2d93b144e458e22 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 14:52:00 UTC) from {username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:40214
I20260927 14:52:00.095217 7871 tablet_replica.cc:330] T 56f8151bb1b14a95a2d93b144e458e22 P a473db7856d24a8ba3efb9bbc9e12de9: stopping tablet replica
I20260927 14:52:00.095363 7871 raft_consensus.cc:2229] T 56f8151bb1b14a95a2d93b144e458e22 P a473db7856d24a8ba3efb9bbc9e12de9 [term 1 LEADER]: Raft consensus shutting down.
I20260927 14:52:00.095679 7871 raft_consensus.cc:2258] T 56f8151bb1b14a95a2d93b144e458e22 P a473db7856d24a8ba3efb9bbc9e12de9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 14:52:00.095759 7414 tablet_service.cc:1514] Processing DeleteTablet for tablet 56f8151bb1b14a95a2d93b144e458e22 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 14:52:00 UTC) from {username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:33956
I20260927 14:52:00.096035 7688 tablet_service.cc:1514] Processing DeleteTablet for tablet 56f8151bb1b14a95a2d93b144e458e22 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 14:52:00 UTC) from {username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:32888
I20260927 14:52:00.096341 7873 tablet_replica.cc:330] T 56f8151bb1b14a95a2d93b144e458e22 P df8799b0da0c4b4e895000a21761211e: stopping tablet replica
I20260927 14:52:00.096339 7871 ts_tablet_manager.cc:1905] T 56f8151bb1b14a95a2d93b144e458e22 P a473db7856d24a8ba3efb9bbc9e12de9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 14:52:00.096504 7873 raft_consensus.cc:2229] T 56f8151bb1b14a95a2d93b144e458e22 P df8799b0da0c4b4e895000a21761211e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 14:52:00.096678 7873 raft_consensus.cc:2258] T 56f8151bb1b14a95a2d93b144e458e22 P df8799b0da0c4b4e895000a21761211e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 14:52:00.097218 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6936m5/build/debug/bin/kudu with pid 7263
I20260927 14:52:00.097574 7873 ts_tablet_manager.cc:1905] T 56f8151bb1b14a95a2d93b144e458e22 P df8799b0da0c4b4e895000a21761211e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 14:52:00.099579 7871 ts_tablet_manager.cc:1918] T 56f8151bb1b14a95a2d93b144e458e22 P a473db7856d24a8ba3efb9bbc9e12de9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 14:52:00.099661 7871 log.cc:1196] T 56f8151bb1b14a95a2d93b144e458e22 P a473db7856d24a8ba3efb9bbc9e12de9: Deleting WAL directory at /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790520684602251-392-0/minicluster-data/ts-1/wal/wals/56f8151bb1b14a95a2d93b144e458e22
I20260927 14:52:00.099939 7871 ts_tablet_manager.cc:1939] T 56f8151bb1b14a95a2d93b144e458e22 P a473db7856d24a8ba3efb9bbc9e12de9: Deleting consensus metadata
I20260927 14:52:00.100489 7873 ts_tablet_manager.cc:1918] T 56f8151bb1b14a95a2d93b144e458e22 P df8799b0da0c4b4e895000a21761211e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 14:52:00.100618 7873 log.cc:1196] T 56f8151bb1b14a95a2d93b144e458e22 P df8799b0da0c4b4e895000a21761211e: Deleting WAL directory at /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790520684602251-392-0/minicluster-data/ts-2/wal/wals/56f8151bb1b14a95a2d93b144e458e22
I20260927 14:52:00.100721 7287 catalog_manager.cc:4841] TS a473db7856d24a8ba3efb9bbc9e12de9 (127.0.98.2:34621): tablet 56f8151bb1b14a95a2d93b144e458e22 (table test-table [id=75b74236ec1646ecb7bbbfc4079a9da6]) successfully deleted
I20260927 14:52:00.101121 7873 ts_tablet_manager.cc:1939] T 56f8151bb1b14a95a2d93b144e458e22 P df8799b0da0c4b4e895000a21761211e: Deleting consensus metadata
I20260927 14:52:00.101836 7286 catalog_manager.cc:4841] TS df8799b0da0c4b4e895000a21761211e (127.0.98.3:38555): tablet 56f8151bb1b14a95a2d93b144e458e22 (table test-table [id=75b74236ec1646ecb7bbbfc4079a9da6]) successfully deleted
I20260927 14:52:00.105371 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6936m5/build/debug/bin/kudu with pid 7488
I20260927 14:52:00.105718 7843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:52:00.105390 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:43017 (local address 127.0.0.1:33970)
0927 14:52:00.105554 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0927 14:52:00.105620 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:43017: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
W20260927 14:52:00.106387 7287 catalog_manager.cc:4584] TS f37a65090f8b4b8f9a4facdb930c9eec (127.0.98.1:43017): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 56f8151bb1b14a95a2d93b144e458e22: Network error: Client connection negotiation failed: client connection to 127.0.98.1:43017: connect: Connection refused (error 111)
I20260927 14:52:00.111519 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6936m5/build/debug/bin/kudu with pid 7624
I20260927 14:52:00.117882 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6936m5/build/debug/bin/kudu with pid 6982
I20260927 14:52:00.123428 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6936m5/build/debug/bin/kudu with pid 7085
2026-09-27T14:52:00Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (5363 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790520684602251-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 14:52:00 dist-test-slave-xk1g krb5kdc[7876](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 14:52:00 dist-test-slave-xk1g krb5kdc[7876](info): set up 2 sockets
Sep 27 14:52:00 dist-test-slave-xk1g krb5kdc[7876](info): commencing operation
krb5kdc: starting...
W20260927 14:52:01.161597 392 mini_kdc.cc:121] Time spent starting KDC: real 1.015s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790520684602251-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790520684602251-392-0/krb5kdc/test-user.keytab.
Sep 27 14:52:01 dist-test-slave-xk1g krb5kdc[7876](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790520721, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T14:52:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T14:52:01Z Disabled control of system clock
WARNING: no policy specified for oryx/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790520684602251-392-0/krb5kdc/oryx_127.0.98.62.keytab.
Entry for principal oryx/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790520684602251-392-0/krb5kdc/oryx_127.0.98.62.keytab.
I20260927 14:52:01.280380 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6936m5/build/debug/bin/kudu
/tmp/dist-test-task6936m5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790520684602251-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790520684602251-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790520684602251-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790520684602251-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42843
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790520684602251-392-0/krb5kdc/oryx_127.0.98.62.keytab
--principal=oryx/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44659
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:42843
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790520684602251-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790520684602251-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790520684602251-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 14:52:01.388303 7889 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 14:52:01.388624 7889 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 14:52:01.388700 7889 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 14:52:01.392731 7889 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 14:52:01.392825 7889 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260927 14:52:01.392856 7889 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 14:52:01.392904 7889 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 14:52:01.392966 7889 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 14:52:01.393005 7889 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 14:52:01.398386 7889 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44659
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790520684602251-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790520684602251-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:42843
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790520684602251-392-0/krb5kdc/oryx_127.0.98.62.keytab
--principal=oryx/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42843
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790520684602251-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7889
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790520684602251-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:15:17 UTC on ccycloud.quasar-ledxfe.root.comops.site
build id 4676134
I20260927 14:52:01.399547 7889 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 14:52:01.400619 7889 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 14:52:01.409701 7894 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 14:52:01.409694 7895 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 14:52:01.409750 7889 server_base.cc:1034] running on GCE node
W20260927 14:52:01.409694 7897 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 14:52:01.410559 7889 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 14:52:01.411737 7889 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 14:52:01.412940 7889 hybrid_clock.cc:648] HybridClock initialized: now 1790520721412921 us; error 33 us; skew 500 ppm
Sep 27 14:52:01 dist-test-slave-xk1g krb5kdc[7876](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790520721, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 14:52:01.415761 7889 init.cc:377] Logged in from keytab as oryx/127.0.98.62@KRBTEST.COM (short username oryx)
I20260927 14:52:01.416711 7889 webserver.cc:460] Webserver started at http://127.0.98.62:35445/ using document root <none> and password file <none>
I20260927 14:52:01.417379 7889 fs_manager.cc:359] Metadata directory not provided
I20260927 14:52:01.417471 7889 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 14:52:01.417711 7889 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 14:52:01.419806 7889 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790520684602251-392-0/minicluster-data/master-0/data/instance:
uuid: "d4bb90ee61a346e783b1407c8f06dc21"
format_stamp: "Formatted at 2026-09-27 14:52:01 on dist-test-slave-xk1g"
server_key: "59b04dcc8179a4a4300d5a044eb6aade"
server_key_iv: "a35f86c9191b2cad1ab37567efbc6824"
server_key_version: "encryptionkey@0"
I20260927 14:52:01.420401 7889 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790520684602251-392-0/minicluster-data/master-0/wal/instance:
uuid: "d4bb90ee61a346e783b1407c8f06dc21"
format_stamp: "Formatted at 2026-09-27 14:52:01 on dist-test-slave-xk1g"
server_key: "59b04dcc8179a4a4300d5a044eb6aade"
server_key_iv: "a35f86c9191b2cad1ab37567efbc6824"
server_key_version: "encryptionkey@0"
I20260927 14:52:01.424233 7889 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260927 14:52:01.426714 7904 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 14:52:01.428016 7889 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s
I20260927 14:52:01.428169 7889 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790520684602251-392-0/minicluster-data/master-0/data,/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790520684602251-392-0/minicluster-data/master-0/wal
uuid: "d4bb90ee61a346e783b1407c8f06dc21"
format_stamp: "Formatted at 2026-09-27 14:52:01 on dist-test-slave-xk1g"
server_key: "59b04dcc8179a4a4300d5a044eb6aade"
server_key_iv: "a35f86c9191b2cad1ab37567efbc6824"
server_key_version: "encryptionkey@0"
I20260927 14:52:01.428289 7889 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790520684602251-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790520684602251-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790520684602251-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 14:52:01.460343 7889 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 14:52:01.463567 7889 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 14:52:01.463785 7889 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 14:52:01.472066 7889 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:42843
I20260927 14:52:01.472087 7966 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:42843 every 8 connection(s)
I20260927 14:52:01.473076 7889 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790520684602251-392-0/minicluster-data/master-0/data/info.pb
I20260927 14:52:01.475847 7967 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 14:52:01.476511 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6936m5/build/debug/bin/kudu as pid 7889
I20260927 14:52:01.476655 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790520684602251-392-0/minicluster-data/master-0/wal/instance
I20260927 14:52:01.477005 392 external_mini_cluster.cc:1428] Setting key 739a67e6ab538e8e1a27702e649c80f4
I20260927 14:52:01.482338 7967 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d4bb90ee61a346e783b1407c8f06dc21: Bootstrap starting.
I20260927 14:52:01.484762 7967 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d4bb90ee61a346e783b1407c8f06dc21: Neither blocks nor log segments found. Creating new log.
I20260927 14:52:01.485700 7967 log.cc:824] T 00000000000000000000000000000000 P d4bb90ee61a346e783b1407c8f06dc21: Log is configured to *not* fsync() on all Append() calls
Sep 27 14:52:01 dist-test-slave-xk1g krb5kdc[7876](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790520721, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260927 14:52:01.487866 7967 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d4bb90ee61a346e783b1407c8f06dc21: No bootstrap required, opened a new log
I20260927 14:52:01.491580 7967 raft_consensus.cc:348] T 00000000000000000000000000000000 P d4bb90ee61a346e783b1407c8f06dc21 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4bb90ee61a346e783b1407c8f06dc21" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42843 } }
I20260927 14:52:01.491820 7967 raft_consensus.cc:374] T 00000000000000000000000000000000 P d4bb90ee61a346e783b1407c8f06dc21 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 14:52:01.491887 7967 raft_consensus.cc:729] T 00000000000000000000000000000000 P d4bb90ee61a346e783b1407c8f06dc21 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d4bb90ee61a346e783b1407c8f06dc21, State: Initialized, Role: FOLLOWER
I20260927 14:52:01.492017 7970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:52:01.478521 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39060 (local address 127.0.98.62:42843)
0927 14:52:01.478971 (+ 450us) server_negotiation.cc:206] Beginning negotiation
0927 14:52:01.478986 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 14:52:01.479021 (+ 35us) server_negotiation.cc:407] Connection header received
0927 14:52:01.479650 (+ 629us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:52:01.479666 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:52:01.480004 (+ 338us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:52:01.480347 (+ 343us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:52:01.481217 (+ 870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:52:01.482177 (+ 960us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:52:01.483085 (+ 908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:52:01.483572 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:52:01.483747 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:52:01.486668 (+ 2921us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:52:01.486673 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:52:01.486687 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 14:52:01.486711 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:52:01.489005 (+ 2294us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:52:01.489575 (+ 570us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:52:01.489580 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:52:01.489584 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:52:01.489688 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:52:01.490004 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:52:01.490006 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:52:01.490007 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:52:01.490379 (+ 372us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:52:01.490534 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0927 14:52:01.490841 (+ 307us) server_negotiation.cc:299] Negotiation successful
0927 14:52:01.491146 (+ 305us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":303,"thread_start_us":149,"threads_started":1}
I20260927 14:52:01.492404 7967 consensus_queue.cc:260] T 00000000000000000000000000000000 P d4bb90ee61a346e783b1407c8f06dc21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4bb90ee61a346e783b1407c8f06dc21" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42843 } }
I20260927 14:52:01.492569 7967 raft_consensus.cc:388] T 00000000000000000000000000000000 P d4bb90ee61a346e783b1407c8f06dc21 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 14:52:01.492661 7967 raft_consensus.cc:482] T 00000000000000000000000000000000 P d4bb90ee61a346e783b1407c8f06dc21 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 14:52:01.492787 7967 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d4bb90ee61a346e783b1407c8f06dc21 [term 0 FOLLOWER]: Advancing to term 1
I20260927 14:52:01.493796 7967 raft_consensus.cc:504] T 00000000000000000000000000000000 P d4bb90ee61a346e783b1407c8f06dc21 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4bb90ee61a346e783b1407c8f06dc21" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42843 } }
I20260927 14:52:01.494148 7967 leader_election.cc:302] T 00000000000000000000000000000000 P d4bb90ee61a346e783b1407c8f06dc21 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d4bb90ee61a346e783b1407c8f06dc21; no voters:
I20260927 14:52:01.494490 7967 leader_election.cc:288] T 00000000000000000000000000000000 P d4bb90ee61a346e783b1407c8f06dc21 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 14:52:01.494624 7972 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d4bb90ee61a346e783b1407c8f06dc21 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 14:52:01.494853 7972 raft_consensus.cc:686] T 00000000000000000000000000000000 P d4bb90ee61a346e783b1407c8f06dc21 [term 1 LEADER]: Becoming Leader. State: Replica: d4bb90ee61a346e783b1407c8f06dc21, State: Running, Role: LEADER
I20260927 14:52:01.495160 7972 consensus_queue.cc:237] T 00000000000000000000000000000000 P d4bb90ee61a346e783b1407c8f06dc21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4bb90ee61a346e783b1407c8f06dc21" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42843 } }
I20260927 14:52:01.495754 7967 sys_catalog.cc:564] T 00000000000000000000000000000000 P d4bb90ee61a346e783b1407c8f06dc21 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 14:52:01.496945 7973 sys_catalog.cc:455] T 00000000000000000000000000000000 P d4bb90ee61a346e783b1407c8f06dc21 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d4bb90ee61a346e783b1407c8f06dc21" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4bb90ee61a346e783b1407c8f06dc21" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42843 } } }
I20260927 14:52:01.497049 7973 sys_catalog.cc:458] T 00000000000000000000000000000000 P d4bb90ee61a346e783b1407c8f06dc21 [sys.catalog]: This master's current role is: LEADER
I20260927 14:52:01.497182 7974 sys_catalog.cc:455] T 00000000000000000000000000000000 P d4bb90ee61a346e783b1407c8f06dc21 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d4bb90ee61a346e783b1407c8f06dc21. Latest consensus state: current_term: 1 leader_uuid: "d4bb90ee61a346e783b1407c8f06dc21" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4bb90ee61a346e783b1407c8f06dc21" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42843 } } }
I20260927 14:52:01.497279 7974 sys_catalog.cc:458] T 00000000000000000000000000000000 P d4bb90ee61a346e783b1407c8f06dc21 [sys.catalog]: This master's current role is: LEADER
I20260927 14:52:01.498828 7981 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 14:52:01.501590 7981 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 14:52:01.508344 7981 catalog_manager.cc:1347] Generated new cluster ID: b0e9232e34aa44a28644af4ede200348
I20260927 14:52:01.508430 7981 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 14:52:01.522877 7981 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 14:52:01.523826 7981 catalog_manager.cc:1504] Loading token signing keys...
I20260927 14:52:01.529358 7981 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d4bb90ee61a346e783b1407c8f06dc21: Generated new TSK 0
I20260927 14:52:01.530017 7981 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790520684602251-392-0/krb5kdc/oryx_127.0.98.1.keytab.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790520684602251-392-0/krb5kdc/oryx_127.0.98.1.keytab.
I20260927 14:52:01.562062 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6936m5/build/debug/bin/kudu
/tmp/dist-test-task6936m5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790520684602251-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790520684602251-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790520684602251-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790520684602251-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42843
--keytab_file=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790520684602251-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44659
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790520684602251-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790520684602251-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790520684602251-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 14:52:01.684119 7993 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 14:52:01.684391 7993 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 14:52:01.684437 7993 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 14:52:01.688789 7993 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 14:52:01.688862 7993 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 14:52:01.688916 7993 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 14:52:01.688961 7993 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 14:52:01.696013 7993 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44659
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790520684602251-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790520684602251-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790520684602251-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790520684602251-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:42843
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7993
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790520684602251-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:15:17 UTC on ccycloud.quasar-ledxfe.root.comops.site
build id 4676134
I20260927 14:52:01.697244 7993 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 14:52:01.698351 7993 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 14:52:01.707767 8001 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 14:52:01.708009 7993 server_base.cc:1034] running on GCE node
W20260927 14:52:01.707792 7998 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 14:52:01.707767 7999 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 14:52:01.708583 7993 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 14:52:01.709307 7993 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 14:52:01.710493 7993 hybrid_clock.cc:648] HybridClock initialized: now 1790520721710481 us; error 27 us; skew 500 ppm
Sep 27 14:52:01 dist-test-slave-xk1g krb5kdc[7876](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790520721, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 14:52:01.713248 7993 init.cc:377] Logged in from keytab as oryx/127.0.98.1@KRBTEST.COM (short username oryx)
I20260927 14:52:01.714102 7993 webserver.cc:460] Webserver started at http://127.0.98.1:40565/ using document root <none> and password file <none>
I20260927 14:52:01.714613 7993 fs_manager.cc:359] Metadata directory not provided
I20260927 14:52:01.714671 7993 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 14:52:01.714838 7993 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 14:52:01.717046 7993 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790520684602251-392-0/minicluster-data/ts-0/data/instance:
uuid: "7a267c5c521e4912aad1c6124b1a92b6"
format_stamp: "Formatted at 2026-09-27 14:52:01 on dist-test-slave-xk1g"
server_key: "97ced950ec731f446b99bdb09c9bdbfa"
server_key_iv: "8b7d6302d603b14d48cb7bb5e9af8e08"
server_key_version: "encryptionkey@0"
I20260927 14:52:01.717830 7993 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790520684602251-392-0/minicluster-data/ts-0/wal/instance:
uuid: "7a267c5c521e4912aad1c6124b1a92b6"
format_stamp: "Formatted at 2026-09-27 14:52:01 on dist-test-slave-xk1g"
server_key: "97ced950ec731f446b99bdb09c9bdbfa"
server_key_iv: "8b7d6302d603b14d48cb7bb5e9af8e08"
server_key_version: "encryptionkey@0"
I20260927 14:52:01.721671 7993 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260927 14:52:01.724191 8008 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 14:52:01.725535 7993 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 14:52:01.725666 7993 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790520684602251-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790520684602251-392-0/minicluster-data/ts-0/wal
uuid: "7a267c5c521e4912aad1c6124b1a92b6"
format_stamp: "Formatted at 2026-09-27 14:52:01 on dist-test-slave-xk1g"
server_key: "97ced950ec731f446b99bdb09c9bdbfa"
server_key_iv: "8b7d6302d603b14d48cb7bb5e9af8e08"
server_key_version: "encryptionkey@0"
I20260927 14:52:01.725787 7993 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790520684602251-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790520684602251-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790520684602251-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 14:52:01.771860 7993 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 14:52:01.774717 7993 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 14:52:01.774919 7993 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 14:52:01.776458 7993 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 14:52:01.776513 7993 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 14:52:01.776576 7993 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 14:52:01.776608 7993 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 27 14:52:01 dist-test-slave-xk1g krb5kdc[7876](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790520721, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260927 14:52:01.787386 7993 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:46477
I20260927 14:52:01.787403 8123 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:46477 every 8 connection(s)
I20260927 14:52:01.788237 7993 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790520684602251-392-0/minicluster-data/ts-0/data/info.pb
I20260927 14:52:01.789619 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6936m5/build/debug/bin/kudu as pid 7993
I20260927 14:52:01.789745 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790520684602251-392-0/minicluster-data/ts-0/wal/instance
I20260927 14:52:01.790045 392 external_mini_cluster.cc:1428] Setting key bde4f37ac659356e41b3979ab6b1f1d0
I20260927 14:52:01.790954 7970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:52:01.777313 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:40899 (local address 127.0.98.62:42843)
0927 14:52:01.777447 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0927 14:52:01.777450 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 14:52:01.778498 (+ 1048us) server_negotiation.cc:407] Connection header received
0927 14:52:01.779587 (+ 1089us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:52:01.779592 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:52:01.779659 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:52:01.779707 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:52:01.781209 (+ 1502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:52:01.781935 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:52:01.783806 (+ 1871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:52:01.784206 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:52:01.784321 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:52:01.787168 (+ 2847us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:52:01.787174 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:52:01.787176 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 14:52:01.787207 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:52:01.788856 (+ 1649us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:52:01.789400 (+ 544us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:52:01.789403 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:52:01.789405 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:52:01.789481 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:52:01.789887 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:52:01.789889 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:52:01.789890 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:52:01.790063 (+ 173us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:52:01.790172 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0927 14:52:01.790651 (+ 479us) server_negotiation.cc:299] Negotiation successful
0927 14:52:01.790784 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260927 14:52:01.791702 8017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:52:01.777874 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42843 (local address 127.0.98.1:40899)
0927 14:52:01.778354 (+ 480us) negotiation.cc:107] Waiting for socket to connect
0927 14:52:01.778395 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0927 14:52:01.779311 (+ 916us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:52:01.780136 (+ 825us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:52:01.780143 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:52:01.780466 (+ 323us) client_negotiation.cc:189] Negotiated authn=SASL
0927 14:52:01.781034 (+ 568us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:52:01.781057 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:52:01.782076 (+ 1019us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:52:01.782080 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:52:01.783648 (+ 1568us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:52:01.783664 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:52:01.784358 (+ 694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:52:01.784362 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:52:01.784557 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:52:01.785206 (+ 649us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 14:52:01.785222 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 14:52:01.786986 (+ 1764us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 14:52:01.788989 (+ 2003us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 14:52:01.788994 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 14:52:01.789008 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0927 14:52:01.789306 (+ 298us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 14:52:01.789589 (+ 283us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 14:52:01.789593 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 14:52:01.789596 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 14:52:01.789787 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 14:52:01.790171 (+ 384us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 14:52:01.790178 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 14:52:01.790418 (+ 240us) client_negotiation.cc:769] Sending connection context
0927 14:52:01.790594 (+ 176us) client_negotiation.cc:240] Negotiation successful
0927 14:52:01.790900 (+ 306us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":294,"thread_start_us":141,"threads_started":1}
I20260927 14:52:01.793010 8124 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42843
I20260927 14:52:01.793171 8124 heartbeater.cc:464] Registering TS with master...
I20260927 14:52:01.793641 8124 heartbeater.cc:511] Master 127.0.98.62:42843 requested a full tablet report, sending...
I20260927 14:52:01.795040 7920 ts_manager.cc:194] Registered new tserver with Master: 7a267c5c521e4912aad1c6124b1a92b6 (127.0.98.1:46477)
I20260927 14:52:01.796571 7920 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:40899
WARNING: no policy specified for oryx/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260927 14:52:01.803088 7970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:52:01.795117 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:41629 (local address 127.0.98.62:42843)
0927 14:52:01.795237 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0927 14:52:01.795241 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 14:52:01.795484 (+ 243us) server_negotiation.cc:407] Connection header received
0927 14:52:01.795594 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:52:01.795597 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:52:01.795637 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:52:01.795717 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:52:01.796323 (+ 606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:52:01.797137 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:52:01.798299 (+ 1162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:52:01.798611 (+ 312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:52:01.798686 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:52:01.799933 (+ 1247us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:52:01.799937 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:52:01.799940 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 14:52:01.799962 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:52:01.801432 (+ 1470us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:52:01.801859 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:52:01.801864 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:52:01.801866 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:52:01.801950 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:52:01.802232 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:52:01.802235 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:52:01.802237 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:52:01.802410 (+ 173us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:52:01.802549 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0927 14:52:01.802807 (+ 258us) server_negotiation.cc:299] Negotiation successful
0927 14:52:01.802911 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":22,"server-negotiator.queue_time_us":42}
I20260927 14:52:01.803095 8131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:52:01.795043 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42843 (local address 127.0.98.1:41629)
0927 14:52:01.795377 (+ 334us) negotiation.cc:107] Waiting for socket to connect
0927 14:52:01.795394 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 14:52:01.795506 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:52:01.795750 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:52:01.795753 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:52:01.795984 (+ 231us) client_negotiation.cc:189] Negotiated authn=SASL
0927 14:52:01.796130 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:52:01.796139 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:52:01.797267 (+ 1128us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:52:01.797270 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:52:01.798164 (+ 894us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:52:01.798174 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:52:01.798831 (+ 657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:52:01.798834 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:52:01.798938 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:52:01.799461 (+ 523us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 14:52:01.799473 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 14:52:01.799788 (+ 315us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 14:52:01.801541 (+ 1753us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 14:52:01.801545 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 14:52:01.801548 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 14:52:01.801759 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 14:52:01.802048 (+ 289us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 14:52:01.802051 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 14:52:01.802053 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 14:52:01.802137 (+ 84us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 14:52:01.802549 (+ 412us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 14:52:01.802553 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 14:52:01.802649 (+ 96us) client_negotiation.cc:769] Sending connection context
0927 14:52:01.802756 (+ 107us) client_negotiation.cc:240] Negotiation successful
0927 14:52:01.802901 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":243,"mutex_wait_us":3,"thread_start_us":96,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790520684602251-392-0/krb5kdc/oryx_127.0.98.2.keytab.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790520684602251-392-0/krb5kdc/oryx_127.0.98.2.keytab.
I20260927 14:52:01.820225 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6936m5/build/debug/bin/kudu
/tmp/dist-test-task6936m5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790520684602251-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790520684602251-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790520684602251-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790520684602251-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42843
--keytab_file=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790520684602251-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44659
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790520684602251-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790520684602251-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790520684602251-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 14:52:01.934440 8133 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 14:52:01.934706 8133 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 14:52:01.934751 8133 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 14:52:01.938952 8133 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 14:52:01.939019 8133 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 14:52:01.939059 8133 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 14:52:01.939092 8133 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 14:52:01.944414 8133 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44659
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790520684602251-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790520684602251-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790520684602251-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790520684602251-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:42843
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8133
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790520684602251-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:15:17 UTC on ccycloud.quasar-ledxfe.root.comops.site
build id 4676134
I20260927 14:52:01.945582 8133 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 14:52:01.946601 8133 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 14:52:01.956166 8139 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 14:52:01.956174 8141 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 14:52:01.956336 8133 server_base.cc:1034] running on GCE node
W20260927 14:52:01.956231 8138 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 14:52:01.956852 8133 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 14:52:01.957510 8133 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 14:52:01.958668 8133 hybrid_clock.cc:648] HybridClock initialized: now 1790520721958652 us; error 31 us; skew 500 ppm
Sep 27 14:52:01 dist-test-slave-xk1g krb5kdc[7876](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790520721, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 14:52:01.961536 8133 init.cc:377] Logged in from keytab as oryx/127.0.98.2@KRBTEST.COM (short username oryx)
I20260927 14:52:01.962430 8133 webserver.cc:460] Webserver started at http://127.0.98.2:40545/ using document root <none> and password file <none>
I20260927 14:52:01.963004 8133 fs_manager.cc:359] Metadata directory not provided
I20260927 14:52:01.963097 8133 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 14:52:01.963330 8133 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 14:52:01.965468 8133 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790520684602251-392-0/minicluster-data/ts-1/data/instance:
uuid: "b9684a38ff254c1287b71a241d319172"
format_stamp: "Formatted at 2026-09-27 14:52:01 on dist-test-slave-xk1g"
server_key: "debb1d0e2516d906e8e8f004c3822ba3"
server_key_iv: "95bd8ec90e558838d550a95f70e7136a"
server_key_version: "encryptionkey@0"
I20260927 14:52:01.966030 8133 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790520684602251-392-0/minicluster-data/ts-1/wal/instance:
uuid: "b9684a38ff254c1287b71a241d319172"
format_stamp: "Formatted at 2026-09-27 14:52:01 on dist-test-slave-xk1g"
server_key: "debb1d0e2516d906e8e8f004c3822ba3"
server_key_iv: "95bd8ec90e558838d550a95f70e7136a"
server_key_version: "encryptionkey@0"
I20260927 14:52:01.969751 8133 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260927 14:52:01.972283 8148 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 14:52:01.973670 8133 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 14:52:01.973845 8133 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790520684602251-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790520684602251-392-0/minicluster-data/ts-1/wal
uuid: "b9684a38ff254c1287b71a241d319172"
format_stamp: "Formatted at 2026-09-27 14:52:01 on dist-test-slave-xk1g"
server_key: "debb1d0e2516d906e8e8f004c3822ba3"
server_key_iv: "95bd8ec90e558838d550a95f70e7136a"
server_key_version: "encryptionkey@0"
I20260927 14:52:01.973966 8133 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790520684602251-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790520684602251-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790520684602251-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 14:52:01.991329 8133 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 14:52:01.994992 8133 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 14:52:01.995196 8133 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 14:52:01.996830 8133 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 14:52:01.996910 8133 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 14:52:01.996978 8133 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 14:52:01.997030 8133 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 27 14:52:02 dist-test-slave-xk1g krb5kdc[7876](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790520721, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260927 14:52:02.008256 8133 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:34999
I20260927 14:52:02.008705 8263 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:34999 every 8 connection(s)
I20260927 14:52:02.009421 8133 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790520684602251-392-0/minicluster-data/ts-1/data/info.pb
I20260927 14:52:02.011616 7970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:52:01.997830 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:59403 (local address 127.0.98.62:42843)
0927 14:52:01.997993 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0927 14:52:01.998000 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 14:52:01.998962 (+ 962us) server_negotiation.cc:407] Connection header received
0927 14:52:02.000199 (+ 1237us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:52:02.000203 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:52:02.000262 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:52:02.000325 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:52:02.002088 (+ 1763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:52:02.002745 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:52:02.003952 (+ 1207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:52:02.004225 (+ 273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:52:02.004345 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:52:02.007571 (+ 3226us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:52:02.007575 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:52:02.007577 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 14:52:02.007604 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:52:02.009567 (+ 1963us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:52:02.010226 (+ 659us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:52:02.010230 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:52:02.010231 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:52:02.010339 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:52:02.010689 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:52:02.010691 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:52:02.010692 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:52:02.010884 (+ 192us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:52:02.010965 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0927 14:52:02.011396 (+ 431us) server_negotiation.cc:299] Negotiation successful
0927 14:52:02.011500 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260927 14:52:02.012463 8157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:52:01.998322 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42843 (local address 127.0.98.2:59403)
0927 14:52:01.998784 (+ 462us) negotiation.cc:107] Waiting for socket to connect
0927 14:52:01.998843 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0927 14:52:01.999835 (+ 992us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:52:02.000818 (+ 983us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:52:02.000830 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:52:02.001369 (+ 539us) client_negotiation.cc:189] Negotiated authn=SASL
0927 14:52:02.001927 (+ 558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:52:02.001942 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:52:02.002882 (+ 940us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:52:02.002886 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:52:02.003815 (+ 929us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:52:02.003825 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:52:02.004409 (+ 584us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:52:02.004413 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:52:02.004631 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:52:02.005358 (+ 727us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 14:52:02.005380 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0927 14:52:02.007352 (+ 1972us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 14:52:02.009704 (+ 2352us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 14:52:02.009709 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 14:52:02.009722 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0927 14:52:02.010086 (+ 364us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 14:52:02.010445 (+ 359us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 14:52:02.010448 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 14:52:02.010450 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 14:52:02.010597 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 14:52:02.010977 (+ 380us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 14:52:02.010982 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 14:52:02.011202 (+ 220us) client_negotiation.cc:769] Sending connection context
0927 14:52:02.011353 (+ 151us) client_negotiation.cc:240] Negotiation successful
0927 14:52:02.011598 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":257,"thread_start_us":122,"threads_started":1}
I20260927 14:52:02.013667 8264 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42843
I20260927 14:52:02.013801 8264 heartbeater.cc:464] Registering TS with master...
I20260927 14:52:02.014353 8264 heartbeater.cc:511] Master 127.0.98.62:42843 requested a full tablet report, sending...
I20260927 14:52:02.015473 7920 ts_manager.cc:194] Registered new tserver with Master: b9684a38ff254c1287b71a241d319172 (127.0.98.2:34999)
I20260927 14:52:02.016261 7920 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:59403
I20260927 14:52:02.017150 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6936m5/build/debug/bin/kudu as pid 8133
I20260927 14:52:02.017261 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790520684602251-392-0/minicluster-data/ts-1/wal/instance
I20260927 14:52:02.017563 392 external_mini_cluster.cc:1428] Setting key f49137240f3cf32cc2c2da2ee9a80189
I20260927 14:52:02.024197 8270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:52:02.015967 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42843 (local address 127.0.98.2:54309)
0927 14:52:02.016271 (+ 304us) negotiation.cc:107] Waiting for socket to connect
0927 14:52:02.016288 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 14:52:02.016375 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:52:02.016835 (+ 460us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:52:02.016837 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:52:02.017099 (+ 262us) client_negotiation.cc:189] Negotiated authn=SASL
0927 14:52:02.017214 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:52:02.017222 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:52:02.018189 (+ 967us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:52:02.018193 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:52:02.018923 (+ 730us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:52:02.018932 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:52:02.019567 (+ 635us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:52:02.019570 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:52:02.019648 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:52:02.020217 (+ 569us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 14:52:02.020229 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 14:52:02.020583 (+ 354us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 14:52:02.022414 (+ 1831us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 14:52:02.022421 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 14:52:02.022426 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 14:52:02.022701 (+ 275us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 14:52:02.023039 (+ 338us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 14:52:02.023046 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 14:52:02.023051 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 14:52:02.023158 (+ 107us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 14:52:02.023656 (+ 498us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 14:52:02.023660 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 14:52:02.023771 (+ 111us) client_negotiation.cc:769] Sending connection context
0927 14:52:02.023887 (+ 116us) client_negotiation.cc:240] Negotiation successful
0927 14:52:02.024044 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":226,"thread_start_us":91,"threads_started":1}
I20260927 14:52:02.024533 7970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:52:02.016050 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:54309 (local address 127.0.98.62:42843)
0927 14:52:02.016627 (+ 577us) server_negotiation.cc:206] Beginning negotiation
0927 14:52:02.016632 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 14:52:02.016643 (+ 11us) server_negotiation.cc:407] Connection header received
0927 14:52:02.016677 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:52:02.016679 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:52:02.016718 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:52:02.016803 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:52:02.017325 (+ 522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:52:02.018035 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:52:02.019034 (+ 999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:52:02.019403 (+ 369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:52:02.019535 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:52:02.020711 (+ 1176us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:52:02.020715 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:52:02.020720 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 14:52:02.020750 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:52:02.022288 (+ 1538us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:52:02.022829 (+ 541us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:52:02.022832 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:52:02.022833 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:52:02.022928 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:52:02.023296 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:52:02.023300 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:52:02.023302 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:52:02.023543 (+ 241us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:52:02.023633 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0927 14:52:02.023987 (+ 354us) server_negotiation.cc:299] Negotiation successful
0927 14:52:02.024138 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":488}
WARNING: no policy specified for oryx/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790520684602251-392-0/krb5kdc/oryx_127.0.98.3.keytab.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790520684602251-392-0/krb5kdc/oryx_127.0.98.3.keytab.
I20260927 14:52:02.049389 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6936m5/build/debug/bin/kudu
/tmp/dist-test-task6936m5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790520684602251-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790520684602251-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790520684602251-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790520684602251-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42843
--keytab_file=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790520684602251-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44659
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790520684602251-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790520684602251-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790520684602251-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 14:52:02.158977 8273 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 14:52:02.159216 8273 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 14:52:02.159260 8273 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 14:52:02.163230 8273 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 14:52:02.163296 8273 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 14:52:02.163336 8273 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 14:52:02.163368 8273 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 14:52:02.168560 8273 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44659
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790520684602251-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790520684602251-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790520684602251-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790520684602251-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:42843
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8273
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790520684602251-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:15:17 UTC on ccycloud.quasar-ledxfe.root.comops.site
build id 4676134
I20260927 14:52:02.169652 8273 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 14:52:02.170943 8273 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 14:52:02.180676 8278 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 14:52:02.180681 8279 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 14:52:02.180716 8281 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 14:52:02.180878 8273 server_base.cc:1034] running on GCE node
I20260927 14:52:02.181381 8273 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 14:52:02.182060 8273 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 14:52:02.183257 8273 hybrid_clock.cc:648] HybridClock initialized: now 1790520722183240 us; error 27 us; skew 500 ppm
Sep 27 14:52:02 dist-test-slave-xk1g krb5kdc[7876](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790520722, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 14:52:02.186090 8273 init.cc:377] Logged in from keytab as oryx/127.0.98.3@KRBTEST.COM (short username oryx)
I20260927 14:52:02.187100 8273 webserver.cc:460] Webserver started at http://127.0.98.3:36297/ using document root <none> and password file <none>
I20260927 14:52:02.187650 8273 fs_manager.cc:359] Metadata directory not provided
I20260927 14:52:02.187784 8273 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 14:52:02.188009 8273 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 14:52:02.190062 8273 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790520684602251-392-0/minicluster-data/ts-2/data/instance:
uuid: "fd36717bc96647548d91ea87dd09dae4"
format_stamp: "Formatted at 2026-09-27 14:52:02 on dist-test-slave-xk1g"
server_key: "ab3d00b671913479706dc8ae65f3b56b"
server_key_iv: "0162d42b28db2978ffcc5aa106d09943"
server_key_version: "encryptionkey@0"
I20260927 14:52:02.190604 8273 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790520684602251-392-0/minicluster-data/ts-2/wal/instance:
uuid: "fd36717bc96647548d91ea87dd09dae4"
format_stamp: "Formatted at 2026-09-27 14:52:02 on dist-test-slave-xk1g"
server_key: "ab3d00b671913479706dc8ae65f3b56b"
server_key_iv: "0162d42b28db2978ffcc5aa106d09943"
server_key_version: "encryptionkey@0"
I20260927 14:52:02.194304 8273 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 14:52:02.196897 8288 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 14:52:02.198298 8273 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260927 14:52:02.198421 8273 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790520684602251-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790520684602251-392-0/minicluster-data/ts-2/wal
uuid: "fd36717bc96647548d91ea87dd09dae4"
format_stamp: "Formatted at 2026-09-27 14:52:02 on dist-test-slave-xk1g"
server_key: "ab3d00b671913479706dc8ae65f3b56b"
server_key_iv: "0162d42b28db2978ffcc5aa106d09943"
server_key_version: "encryptionkey@0"
I20260927 14:52:02.198536 8273 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790520684602251-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790520684602251-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790520684602251-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 14:52:02.227250 8273 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 14:52:02.230181 8273 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 14:52:02.230377 8273 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 14:52:02.231858 8273 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 14:52:02.231926 8273 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 14:52:02.231994 8273 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 14:52:02.232033 8273 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 27 14:52:02 dist-test-slave-xk1g krb5kdc[7876](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790520722, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260927 14:52:02.243072 8273 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:43169
I20260927 14:52:02.243146 8403 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:43169 every 8 connection(s)
I20260927 14:52:02.243953 8273 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790520684602251-392-0/minicluster-data/ts-2/data/info.pb
I20260927 14:52:02.245425 7970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:52:02.232577 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:45559 (local address 127.0.98.62:42843)
0927 14:52:02.232696 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0927 14:52:02.232701 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 14:52:02.233759 (+ 1058us) server_negotiation.cc:407] Connection header received
0927 14:52:02.234817 (+ 1058us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:52:02.234820 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:52:02.234907 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:52:02.234970 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:52:02.236423 (+ 1453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:52:02.236992 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:52:02.238471 (+ 1479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:52:02.238761 (+ 290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:52:02.238881 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:52:02.241616 (+ 2735us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:52:02.241620 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:52:02.241624 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 14:52:02.241654 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:52:02.243386 (+ 1732us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:52:02.243894 (+ 508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:52:02.243899 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:52:02.243901 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:52:02.243988 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:52:02.244320 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:52:02.244325 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:52:02.244327 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:52:02.244552 (+ 225us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:52:02.244651 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0927 14:52:02.245176 (+ 525us) server_negotiation.cc:299] Negotiation successful
0927 14:52:02.245300 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260927 14:52:02.245950 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6936m5/build/debug/bin/kudu as pid 8273
I20260927 14:52:02.246104 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790520684602251-392-0/minicluster-data/ts-2/wal/instance
I20260927 14:52:02.246373 392 external_mini_cluster.cc:1428] Setting key 81172a9c5bbb1e535a47e2844fd99f41
I20260927 14:52:02.246426 8297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:52:02.233126 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42843 (local address 127.0.98.3:45559)
0927 14:52:02.233606 (+ 480us) negotiation.cc:107] Waiting for socket to connect
0927 14:52:02.233657 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0927 14:52:02.234569 (+ 912us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:52:02.235438 (+ 869us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:52:02.235445 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:52:02.235824 (+ 379us) client_negotiation.cc:189] Negotiated authn=SASL
0927 14:52:02.236280 (+ 456us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:52:02.236292 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:52:02.237162 (+ 870us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:52:02.237165 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:52:02.238342 (+ 1177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:52:02.238351 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:52:02.238928 (+ 577us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:52:02.238932 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:52:02.239110 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:52:02.239764 (+ 654us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 14:52:02.239788 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0927 14:52:02.241446 (+ 1658us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 14:52:02.243519 (+ 2073us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 14:52:02.243525 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 14:52:02.243538 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0927 14:52:02.243789 (+ 251us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 14:52:02.244080 (+ 291us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 14:52:02.244085 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 14:52:02.244087 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 14:52:02.244224 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 14:52:02.244664 (+ 440us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 14:52:02.244670 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 14:52:02.244910 (+ 240us) client_negotiation.cc:769] Sending connection context
0927 14:52:02.245117 (+ 207us) client_negotiation.cc:240] Negotiation successful
0927 14:52:02.245436 (+ 319us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":296,"thread_start_us":131,"threads_started":1}
I20260927 14:52:02.247731 8404 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42843
I20260927 14:52:02.247898 8404 heartbeater.cc:464] Registering TS with master...
I20260927 14:52:02.248485 8404 heartbeater.cc:511] Master 127.0.98.62:42843 requested a full tablet report, sending...
I20260927 14:52:02.249503 7920 ts_manager.cc:194] Registered new tserver with Master: fd36717bc96647548d91ea87dd09dae4 (127.0.98.3:43169)
I20260927 14:52:02.250177 7920 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:45559
I20260927 14:52:02.256441 7970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:52:02.249907 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:54909 (local address 127.0.98.62:42843)
0927 14:52:02.250022 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0927 14:52:02.250027 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 14:52:02.250132 (+ 105us) server_negotiation.cc:407] Connection header received
0927 14:52:02.250281 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:52:02.250284 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:52:02.250327 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:52:02.250415 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:52:02.250851 (+ 436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:52:02.251366 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:52:02.252192 (+ 826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:52:02.252485 (+ 293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:52:02.252597 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:52:02.253691 (+ 1094us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:52:02.253695 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:52:02.253697 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 14:52:02.253721 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:52:02.254879 (+ 1158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:52:02.255304 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:52:02.255308 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:52:02.255310 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:52:02.255388 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:52:02.255657 (+ 269us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:52:02.255660 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:52:02.255661 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:52:02.255831 (+ 170us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:52:02.255930 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0927 14:52:02.256181 (+ 251us) server_negotiation.cc:299] Negotiation successful
0927 14:52:02.256305 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260927 14:52:02.256453 8410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:52:02.249783 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42843 (local address 127.0.98.3:54909)
0927 14:52:02.250035 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0927 14:52:02.250051 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 14:52:02.250172 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:52:02.250430 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:52:02.250432 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:52:02.250648 (+ 216us) client_negotiation.cc:189] Negotiated authn=SASL
0927 14:52:02.250738 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:52:02.250743 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:52:02.251492 (+ 749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:52:02.251496 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:52:02.252056 (+ 560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:52:02.252063 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:52:02.252600 (+ 537us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:52:02.252604 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:52:02.252693 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:52:02.253203 (+ 510us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 14:52:02.253229 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0927 14:52:02.253542 (+ 313us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 14:52:02.254990 (+ 1448us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 14:52:02.254994 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 14:52:02.254997 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 14:52:02.255200 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 14:52:02.255477 (+ 277us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 14:52:02.255480 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 14:52:02.255482 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 14:52:02.255561 (+ 79us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 14:52:02.255930 (+ 369us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 14:52:02.255934 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 14:52:02.256022 (+ 88us) client_negotiation.cc:769] Sending connection context
0927 14:52:02.256120 (+ 98us) client_negotiation.cc:240] Negotiation successful
0927 14:52:02.256277 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":91,"threads_started":1}
I20260927 14:52:02.260049 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 27 14:52:02 dist-test-slave-xk1g krb5kdc[7876](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790520722, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 27 14:52:02 dist-test-slave-xk1g krb5kdc[7876](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 14:52:02 dist-test-slave-xk1g krb5kdc[7876](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
W20260927 14:52:02.282871 7970 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 14:52:02.276871 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39068 (local address 127.0.98.62:42843)
0927 14:52:02.276994 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0927 14:52:02.276997 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 14:52:02.277009 (+ 12us) server_negotiation.cc:407] Connection header received
0927 14:52:02.277049 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:52:02.277051 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:52:02.277150 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:52:02.277222 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:52:02.277910 (+ 688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:52:02.278505 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:52:02.279275 (+ 770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:52:02.279478 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:52:02.279550 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:52:02.282669 (+ 3119us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39068: BlockingRecv error: recv got EOF from 127.0.0.1:39068 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
Sep 27 14:52:02 dist-test-slave-xk1g krb5kdc[7876](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790520722, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260927 14:52:02.291771 7970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:52:02.284126 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39078 (local address 127.0.98.62:42843)
0927 14:52:02.284257 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0927 14:52:02.284260 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 14:52:02.284299 (+ 39us) server_negotiation.cc:407] Connection header received
0927 14:52:02.284423 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:52:02.284425 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:52:02.284459 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:52:02.284523 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:52:02.285043 (+ 520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:52:02.285569 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:52:02.286340 (+ 771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:52:02.286564 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:52:02.286670 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:52:02.288845 (+ 2175us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:52:02.288848 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:52:02.288850 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 14:52:02.288884 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:52:02.290274 (+ 1390us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:52:02.290673 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:52:02.290675 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:52:02.290677 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:52:02.290741 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:52:02.290954 (+ 213us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:52:02.290955 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:52:02.290956 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:52:02.291100 (+ 144us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:52:02.291194 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0927 14:52:02.291419 (+ 225us) server_negotiation.cc:299] Negotiation successful
0927 14:52:02.291573 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260927 14:52:02.294726 7920 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39078:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 14:52:02.298077 7920 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 14:52:02.312374 8429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:52:02.306556 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:34999 (local address 127.0.0.1:52474)
0927 14:52:02.307440 (+ 884us) negotiation.cc:107] Waiting for socket to connect
0927 14:52:02.307456 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 14:52:02.307614 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:52:02.308241 (+ 627us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:52:02.308244 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:52:02.308257 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 14:52:02.308355 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:52:02.308360 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:52:02.309587 (+ 1227us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:52:02.309589 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:52:02.310991 (+ 1402us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:52:02.311002 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:52:02.311822 (+ 820us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:52:02.311826 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:52:02.311939 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:52:02.311981 (+ 42us) client_negotiation.cc:769] Sending connection context
0927 14:52:02.312123 (+ 142us) client_negotiation.cc:240] Negotiation successful
0927 14:52:02.312237 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":809,"mutex_wait_us":90,"thread_start_us":100,"threads_started":1}
I20260927 14:52:02.312384 8428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:52:02.306947 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52474 (local address 127.0.98.2:34999)
0927 14:52:02.307256 (+ 309us) server_negotiation.cc:206] Beginning negotiation
0927 14:52:02.307277 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0927 14:52:02.307545 (+ 268us) server_negotiation.cc:407] Connection header received
0927 14:52:02.307761 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:52:02.307767 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:52:02.307957 (+ 190us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:52:02.308082 (+ 125us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 14:52:02.308458 (+ 376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:52:02.309482 (+ 1024us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:52:02.311144 (+ 1662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:52:02.311697 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:52:02.311823 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:52:02.311990 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0927 14:52:02.312153 (+ 163us) server_negotiation.cc:299] Negotiation successful
0927 14:52:02.312243 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":213,"thread_start_us":95,"threads_started":1}
I20260927 14:52:02.312722 8426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:52:02.306265 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:43169 (local address 127.0.0.1:46842)
0927 14:52:02.307079 (+ 814us) negotiation.cc:107] Waiting for socket to connect
0927 14:52:02.307139 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0927 14:52:02.307271 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:52:02.308135 (+ 864us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:52:02.308140 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:52:02.308160 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 14:52:02.308282 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:52:02.308288 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:52:02.309420 (+ 1132us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:52:02.309423 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:52:02.310999 (+ 1576us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:52:02.311004 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:52:02.312202 (+ 1198us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:52:02.312206 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:52:02.312358 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:52:02.312456 (+ 98us) client_negotiation.cc:769] Sending connection context
0927 14:52:02.312527 (+ 71us) client_negotiation.cc:240] Negotiation successful
0927 14:52:02.312597 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":639,"mutex_wait_us":160,"thread_start_us":60,"threads_started":1}
I20260927 14:52:02.312824 8430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:52:02.307225 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56588 (local address 127.0.98.1:46477)
0927 14:52:02.307584 (+ 359us) server_negotiation.cc:206] Beginning negotiation
0927 14:52:02.307590 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 14:52:02.307879 (+ 289us) server_negotiation.cc:407] Connection header received
0927 14:52:02.308097 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:52:02.308103 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:52:02.308270 (+ 167us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:52:02.308432 (+ 162us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 14:52:02.308784 (+ 352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:52:02.309755 (+ 971us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:52:02.311176 (+ 1421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:52:02.311921 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:52:02.312001 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:52:02.312128 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0927 14:52:02.312585 (+ 457us) server_negotiation.cc:299] Negotiation successful
0927 14:52:02.312677 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":267,"thread_start_us":80,"threads_started":1}
I20260927 14:52:02.313203 8431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:52:02.306976 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:46477 (local address 127.0.0.1:56588)
0927 14:52:02.307797 (+ 821us) negotiation.cc:107] Waiting for socket to connect
0927 14:52:02.307809 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 14:52:02.307937 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:52:02.308565 (+ 628us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:52:02.308568 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:52:02.308585 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 14:52:02.308661 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:52:02.308666 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:52:02.309856 (+ 1190us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:52:02.309861 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:52:02.311023 (+ 1162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:52:02.311032 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:52:02.312210 (+ 1178us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:52:02.312213 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:52:02.312363 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:52:02.312458 (+ 95us) client_negotiation.cc:769] Sending connection context
0927 14:52:02.312988 (+ 530us) client_negotiation.cc:240] Negotiation successful
0927 14:52:02.313079 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":743,"mutex_wait_us":192,"thread_start_us":496,"threads_started":1}
I20260927 14:52:02.313299 8427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:52:02.306408 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46842 (local address 127.0.98.3:43169)
0927 14:52:02.306776 (+ 368us) server_negotiation.cc:206] Beginning negotiation
0927 14:52:02.306790 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 14:52:02.307428 (+ 638us) server_negotiation.cc:407] Connection header received
0927 14:52:02.307529 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:52:02.307534 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:52:02.307741 (+ 207us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:52:02.308141 (+ 400us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 14:52:02.308394 (+ 253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:52:02.309310 (+ 916us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:52:02.311114 (+ 1804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:52:02.311916 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:52:02.312015 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:52:02.312143 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0927 14:52:02.312979 (+ 836us) server_negotiation.cc:299] Negotiation successful
0927 14:52:02.313100 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":254,"thread_start_us":164,"threads_started":1}
I20260927 14:52:02.315587 8198 tablet_service.cc:1467] Processing CreateTablet for tablet da8422bca7c54af4af049dc8c472fc8a (DEFAULT_TABLE table=test-table [id=d7c8361fa70343b9bc7954252aa9cf0c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 14:52:02.315735 8058 tablet_service.cc:1467] Processing CreateTablet for tablet da8422bca7c54af4af049dc8c472fc8a (DEFAULT_TABLE table=test-table [id=d7c8361fa70343b9bc7954252aa9cf0c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 14:52:02.316542 8198 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet da8422bca7c54af4af049dc8c472fc8a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 14:52:02.316629 8058 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet da8422bca7c54af4af049dc8c472fc8a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 14:52:02.318857 8338 tablet_service.cc:1467] Processing CreateTablet for tablet da8422bca7c54af4af049dc8c472fc8a (DEFAULT_TABLE table=test-table [id=d7c8361fa70343b9bc7954252aa9cf0c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 14:52:02.319831 8338 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet da8422bca7c54af4af049dc8c472fc8a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 14:52:02.323344 8432 tablet_bootstrap.cc:492] T da8422bca7c54af4af049dc8c472fc8a P 7a267c5c521e4912aad1c6124b1a92b6: Bootstrap starting.
I20260927 14:52:02.323769 8433 tablet_bootstrap.cc:492] T da8422bca7c54af4af049dc8c472fc8a P b9684a38ff254c1287b71a241d319172: Bootstrap starting.
I20260927 14:52:02.325600 8432 tablet_bootstrap.cc:654] T da8422bca7c54af4af049dc8c472fc8a P 7a267c5c521e4912aad1c6124b1a92b6: Neither blocks nor log segments found. Creating new log.
I20260927 14:52:02.326090 8433 tablet_bootstrap.cc:654] T da8422bca7c54af4af049dc8c472fc8a P b9684a38ff254c1287b71a241d319172: Neither blocks nor log segments found. Creating new log.
I20260927 14:52:02.326395 8434 tablet_bootstrap.cc:492] T da8422bca7c54af4af049dc8c472fc8a P fd36717bc96647548d91ea87dd09dae4: Bootstrap starting.
I20260927 14:52:02.326493 8432 log.cc:824] T da8422bca7c54af4af049dc8c472fc8a P 7a267c5c521e4912aad1c6124b1a92b6: Log is configured to *not* fsync() on all Append() calls
I20260927 14:52:02.326874 8433 log.cc:824] T da8422bca7c54af4af049dc8c472fc8a P b9684a38ff254c1287b71a241d319172: Log is configured to *not* fsync() on all Append() calls
I20260927 14:52:02.328467 8434 tablet_bootstrap.cc:654] T da8422bca7c54af4af049dc8c472fc8a P fd36717bc96647548d91ea87dd09dae4: Neither blocks nor log segments found. Creating new log.
I20260927 14:52:02.328619 8432 tablet_bootstrap.cc:492] T da8422bca7c54af4af049dc8c472fc8a P 7a267c5c521e4912aad1c6124b1a92b6: No bootstrap required, opened a new log
I20260927 14:52:02.328840 8433 tablet_bootstrap.cc:492] T da8422bca7c54af4af049dc8c472fc8a P b9684a38ff254c1287b71a241d319172: No bootstrap required, opened a new log
I20260927 14:52:02.328848 8432 ts_tablet_manager.cc:1397] T da8422bca7c54af4af049dc8c472fc8a P 7a267c5c521e4912aad1c6124b1a92b6: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260927 14:52:02.329018 8433 ts_tablet_manager.cc:1397] T da8422bca7c54af4af049dc8c472fc8a P b9684a38ff254c1287b71a241d319172: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260927 14:52:02.329306 8434 log.cc:824] T da8422bca7c54af4af049dc8c472fc8a P fd36717bc96647548d91ea87dd09dae4: Log is configured to *not* fsync() on all Append() calls
I20260927 14:52:02.331182 8434 tablet_bootstrap.cc:492] T da8422bca7c54af4af049dc8c472fc8a P fd36717bc96647548d91ea87dd09dae4: No bootstrap required, opened a new log
I20260927 14:52:02.331405 8434 ts_tablet_manager.cc:1397] T da8422bca7c54af4af049dc8c472fc8a P fd36717bc96647548d91ea87dd09dae4: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260927 14:52:02.332734 8432 raft_consensus.cc:348] T da8422bca7c54af4af049dc8c472fc8a P 7a267c5c521e4912aad1c6124b1a92b6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd36717bc96647548d91ea87dd09dae4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43169 } } peers { permanent_uuid: "7a267c5c521e4912aad1c6124b1a92b6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46477 } } peers { permanent_uuid: "b9684a38ff254c1287b71a241d319172" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34999 } }
I20260927 14:52:02.333006 8432 raft_consensus.cc:374] T da8422bca7c54af4af049dc8c472fc8a P 7a267c5c521e4912aad1c6124b1a92b6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 14:52:02.332902 8433 raft_consensus.cc:348] T da8422bca7c54af4af049dc8c472fc8a P b9684a38ff254c1287b71a241d319172 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd36717bc96647548d91ea87dd09dae4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43169 } } peers { permanent_uuid: "7a267c5c521e4912aad1c6124b1a92b6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46477 } } peers { permanent_uuid: "b9684a38ff254c1287b71a241d319172" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34999 } }
I20260927 14:52:02.333060 8432 raft_consensus.cc:729] T da8422bca7c54af4af049dc8c472fc8a P 7a267c5c521e4912aad1c6124b1a92b6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a267c5c521e4912aad1c6124b1a92b6, State: Initialized, Role: FOLLOWER
I20260927 14:52:02.333161 8433 raft_consensus.cc:374] T da8422bca7c54af4af049dc8c472fc8a P b9684a38ff254c1287b71a241d319172 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 14:52:02.333223 8433 raft_consensus.cc:729] T da8422bca7c54af4af049dc8c472fc8a P b9684a38ff254c1287b71a241d319172 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b9684a38ff254c1287b71a241d319172, State: Initialized, Role: FOLLOWER
I20260927 14:52:02.333536 8432 consensus_queue.cc:260] T da8422bca7c54af4af049dc8c472fc8a P 7a267c5c521e4912aad1c6124b1a92b6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd36717bc96647548d91ea87dd09dae4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43169 } } peers { permanent_uuid: "7a267c5c521e4912aad1c6124b1a92b6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46477 } } peers { permanent_uuid: "b9684a38ff254c1287b71a241d319172" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34999 } }
I20260927 14:52:02.333627 8433 consensus_queue.cc:260] T da8422bca7c54af4af049dc8c472fc8a P b9684a38ff254c1287b71a241d319172 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd36717bc96647548d91ea87dd09dae4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43169 } } peers { permanent_uuid: "7a267c5c521e4912aad1c6124b1a92b6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46477 } } peers { permanent_uuid: "b9684a38ff254c1287b71a241d319172" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34999 } }
I20260927 14:52:02.334092 8124 heartbeater.cc:503] Master 127.0.98.62:42843 was elected leader, sending a full tablet report...
I20260927 14:52:02.334425 8432 ts_tablet_manager.cc:1428] T da8422bca7c54af4af049dc8c472fc8a P 7a267c5c521e4912aad1c6124b1a92b6: Time spent starting tablet: real 0.005s user 0.003s sys 0.003s
I20260927 14:52:02.334489 8433 ts_tablet_manager.cc:1428] T da8422bca7c54af4af049dc8c472fc8a P b9684a38ff254c1287b71a241d319172: Time spent starting tablet: real 0.005s user 0.002s sys 0.003s
I20260927 14:52:02.334774 8264 heartbeater.cc:503] Master 127.0.98.62:42843 was elected leader, sending a full tablet report...
I20260927 14:52:02.335088 8434 raft_consensus.cc:348] T da8422bca7c54af4af049dc8c472fc8a P fd36717bc96647548d91ea87dd09dae4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd36717bc96647548d91ea87dd09dae4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43169 } } peers { permanent_uuid: "7a267c5c521e4912aad1c6124b1a92b6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46477 } } peers { permanent_uuid: "b9684a38ff254c1287b71a241d319172" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34999 } }
I20260927 14:52:02.335325 8434 raft_consensus.cc:374] T da8422bca7c54af4af049dc8c472fc8a P fd36717bc96647548d91ea87dd09dae4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 14:52:02.335381 8434 raft_consensus.cc:729] T da8422bca7c54af4af049dc8c472fc8a P fd36717bc96647548d91ea87dd09dae4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd36717bc96647548d91ea87dd09dae4, State: Initialized, Role: FOLLOWER
I20260927 14:52:02.335767 8434 consensus_queue.cc:260] T da8422bca7c54af4af049dc8c472fc8a P fd36717bc96647548d91ea87dd09dae4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd36717bc96647548d91ea87dd09dae4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43169 } } peers { permanent_uuid: "7a267c5c521e4912aad1c6124b1a92b6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46477 } } peers { permanent_uuid: "b9684a38ff254c1287b71a241d319172" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34999 } }
I20260927 14:52:02.336360 8404 heartbeater.cc:503] Master 127.0.98.62:42843 was elected leader, sending a full tablet report...
I20260927 14:52:02.336673 8434 ts_tablet_manager.cc:1428] T da8422bca7c54af4af049dc8c472fc8a P fd36717bc96647548d91ea87dd09dae4: Time spent starting tablet: real 0.005s user 0.000s sys 0.005s
W20260927 14:52:02.494882 8405 tablet.cc:2369] T da8422bca7c54af4af049dc8c472fc8a P fd36717bc96647548d91ea87dd09dae4: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 14:52:02.510667 8265 tablet.cc:2369] T da8422bca7c54af4af049dc8c472fc8a P b9684a38ff254c1287b71a241d319172: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 14:52:02.539673 8125 tablet.cc:2369] T da8422bca7c54af4af049dc8c472fc8a P 7a267c5c521e4912aad1c6124b1a92b6: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 14:52:02.563367 8438 raft_consensus.cc:482] T da8422bca7c54af4af049dc8c472fc8a P 7a267c5c521e4912aad1c6124b1a92b6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 14:52:02.563588 8438 raft_consensus.cc:504] T da8422bca7c54af4af049dc8c472fc8a P 7a267c5c521e4912aad1c6124b1a92b6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd36717bc96647548d91ea87dd09dae4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43169 } } peers { permanent_uuid: "7a267c5c521e4912aad1c6124b1a92b6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46477 } } peers { permanent_uuid: "b9684a38ff254c1287b71a241d319172" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34999 } }
I20260927 14:52:02.564812 8438 leader_election.cc:288] T da8422bca7c54af4af049dc8c472fc8a P 7a267c5c521e4912aad1c6124b1a92b6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fd36717bc96647548d91ea87dd09dae4 (127.0.98.3:43169), b9684a38ff254c1287b71a241d319172 (127.0.98.2:34999)
I20260927 14:52:02.570519 8428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:52:02.565174 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:52903 (local address 127.0.98.2:34999)
0927 14:52:02.565400 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0927 14:52:02.565406 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 14:52:02.565678 (+ 272us) server_negotiation.cc:407] Connection header received
0927 14:52:02.565860 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:52:02.565864 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:52:02.565935 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:52:02.566011 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 14:52:02.566466 (+ 455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:52:02.567256 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:52:02.569231 (+ 1975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:52:02.569855 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:52:02.569974 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:52:02.570037 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0927 14:52:02.570317 (+ 280us) server_negotiation.cc:299] Negotiation successful
0927 14:52:02.570392 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":106}
I20260927 14:52:02.570519 8427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:52:02.565824 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:33289 (local address 127.0.98.3:43169)
0927 14:52:02.565985 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0927 14:52:02.565992 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 14:52:02.566010 (+ 18us) server_negotiation.cc:407] Connection header received
0927 14:52:02.566071 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:52:02.566076 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:52:02.566134 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:52:02.566221 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 14:52:02.566488 (+ 267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:52:02.567248 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:52:02.569219 (+ 1971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:52:02.569871 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:52:02.569975 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:52:02.570037 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0927 14:52:02.570317 (+ 280us) server_negotiation.cc:299] Negotiation successful
0927 14:52:02.570391 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260927 14:52:02.570565 8441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:52:02.565034 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:43169 (local address 127.0.98.1:33289)
0927 14:52:02.565536 (+ 502us) negotiation.cc:107] Waiting for socket to connect
0927 14:52:02.565551 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 14:52:02.565684 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:52:02.566229 (+ 545us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:52:02.566233 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:52:02.566247 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 14:52:02.566378 (+ 131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:52:02.566387 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:52:02.567368 (+ 981us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:52:02.567371 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:52:02.569029 (+ 1658us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:52:02.569052 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:52:02.569994 (+ 942us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:52:02.569998 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:52:02.570119 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:52:02.570195 (+ 76us) client_negotiation.cc:769] Sending connection context
0927 14:52:02.570274 (+ 79us) client_negotiation.cc:240] Negotiation successful
0927 14:52:02.570373 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":428,"mutex_wait_us":320,"thread_start_us":190,"threads_started":1}
I20260927 14:52:02.570565 8442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:52:02.565123 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:34999 (local address 127.0.98.1:52903)
0927 14:52:02.565575 (+ 452us) negotiation.cc:107] Waiting for socket to connect
0927 14:52:02.565593 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 14:52:02.565718 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:52:02.566045 (+ 327us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:52:02.566048 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:52:02.566063 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 14:52:02.566311 (+ 248us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:52:02.566319 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:52:02.567368 (+ 1049us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:52:02.567371 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:52:02.569029 (+ 1658us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:52:02.569052 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:52:02.569994 (+ 942us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:52:02.569998 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:52:02.570123 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:52:02.570200 (+ 77us) client_negotiation.cc:769] Sending connection context
0927 14:52:02.570274 (+ 74us) client_negotiation.cc:240] Negotiation successful
0927 14:52:02.570373 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":367,"mutex_wait_us":213,"thread_start_us":117,"threads_started":1}
I20260927 14:52:02.571259 8358 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da8422bca7c54af4af049dc8c472fc8a" candidate_uuid: "7a267c5c521e4912aad1c6124b1a92b6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd36717bc96647548d91ea87dd09dae4" is_pre_election: true
I20260927 14:52:02.571338 8218 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da8422bca7c54af4af049dc8c472fc8a" candidate_uuid: "7a267c5c521e4912aad1c6124b1a92b6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9684a38ff254c1287b71a241d319172" is_pre_election: true
I20260927 14:52:02.571547 8358 raft_consensus.cc:2454] T da8422bca7c54af4af049dc8c472fc8a P fd36717bc96647548d91ea87dd09dae4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a267c5c521e4912aad1c6124b1a92b6 in term 0.
I20260927 14:52:02.571656 8218 raft_consensus.cc:2454] T da8422bca7c54af4af049dc8c472fc8a P b9684a38ff254c1287b71a241d319172 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a267c5c521e4912aad1c6124b1a92b6 in term 0.
I20260927 14:52:02.572100 8009 leader_election.cc:302] T da8422bca7c54af4af049dc8c472fc8a P 7a267c5c521e4912aad1c6124b1a92b6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 7a267c5c521e4912aad1c6124b1a92b6, b9684a38ff254c1287b71a241d319172, fd36717bc96647548d91ea87dd09dae4; no voters:
I20260927 14:52:02.572340 8438 raft_consensus.cc:2781] T da8422bca7c54af4af049dc8c472fc8a P 7a267c5c521e4912aad1c6124b1a92b6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 14:52:02.572418 8438 raft_consensus.cc:482] T da8422bca7c54af4af049dc8c472fc8a P 7a267c5c521e4912aad1c6124b1a92b6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 14:52:02.572463 8438 raft_consensus.cc:3037] T da8422bca7c54af4af049dc8c472fc8a P 7a267c5c521e4912aad1c6124b1a92b6 [term 0 FOLLOWER]: Advancing to term 1
I20260927 14:52:02.573490 8438 raft_consensus.cc:504] T da8422bca7c54af4af049dc8c472fc8a P 7a267c5c521e4912aad1c6124b1a92b6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd36717bc96647548d91ea87dd09dae4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43169 } } peers { permanent_uuid: "7a267c5c521e4912aad1c6124b1a92b6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46477 } } peers { permanent_uuid: "b9684a38ff254c1287b71a241d319172" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34999 } }
I20260927 14:52:02.573848 8438 leader_election.cc:288] T da8422bca7c54af4af049dc8c472fc8a P 7a267c5c521e4912aad1c6124b1a92b6 [CANDIDATE]: Term 1 election: Requested vote from peers fd36717bc96647548d91ea87dd09dae4 (127.0.98.3:43169), b9684a38ff254c1287b71a241d319172 (127.0.98.2:34999)
I20260927 14:52:02.574146 8358 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da8422bca7c54af4af049dc8c472fc8a" candidate_uuid: "7a267c5c521e4912aad1c6124b1a92b6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd36717bc96647548d91ea87dd09dae4"
I20260927 14:52:02.574214 8218 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da8422bca7c54af4af049dc8c472fc8a" candidate_uuid: "7a267c5c521e4912aad1c6124b1a92b6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9684a38ff254c1287b71a241d319172"
I20260927 14:52:02.574257 8358 raft_consensus.cc:3037] T da8422bca7c54af4af049dc8c472fc8a P fd36717bc96647548d91ea87dd09dae4 [term 0 FOLLOWER]: Advancing to term 1
I20260927 14:52:02.574353 8218 raft_consensus.cc:3037] T da8422bca7c54af4af049dc8c472fc8a P b9684a38ff254c1287b71a241d319172 [term 0 FOLLOWER]: Advancing to term 1
I20260927 14:52:02.575358 8218 raft_consensus.cc:2454] T da8422bca7c54af4af049dc8c472fc8a P b9684a38ff254c1287b71a241d319172 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a267c5c521e4912aad1c6124b1a92b6 in term 1.
I20260927 14:52:02.575676 8012 leader_election.cc:302] T da8422bca7c54af4af049dc8c472fc8a P 7a267c5c521e4912aad1c6124b1a92b6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a267c5c521e4912aad1c6124b1a92b6, b9684a38ff254c1287b71a241d319172; no voters:
I20260927 14:52:02.575726 8358 raft_consensus.cc:2454] T da8422bca7c54af4af049dc8c472fc8a P fd36717bc96647548d91ea87dd09dae4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a267c5c521e4912aad1c6124b1a92b6 in term 1.
I20260927 14:52:02.575836 8438 raft_consensus.cc:2781] T da8422bca7c54af4af049dc8c472fc8a P 7a267c5c521e4912aad1c6124b1a92b6 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 14:52:02.576047 8438 raft_consensus.cc:686] T da8422bca7c54af4af049dc8c472fc8a P 7a267c5c521e4912aad1c6124b1a92b6 [term 1 LEADER]: Becoming Leader. State: Replica: 7a267c5c521e4912aad1c6124b1a92b6, State: Running, Role: LEADER
I20260927 14:52:02.576366 8438 consensus_queue.cc:237] T da8422bca7c54af4af049dc8c472fc8a P 7a267c5c521e4912aad1c6124b1a92b6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd36717bc96647548d91ea87dd09dae4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43169 } } peers { permanent_uuid: "7a267c5c521e4912aad1c6124b1a92b6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46477 } } peers { permanent_uuid: "b9684a38ff254c1287b71a241d319172" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34999 } }
I20260927 14:52:02.579623 7919 catalog_manager.cc:5495] T da8422bca7c54af4af049dc8c472fc8a P 7a267c5c521e4912aad1c6124b1a92b6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7a267c5c521e4912aad1c6124b1a92b6 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "7a267c5c521e4912aad1c6124b1a92b6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd36717bc96647548d91ea87dd09dae4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43169 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7a267c5c521e4912aad1c6124b1a92b6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46477 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b9684a38ff254c1287b71a241d319172" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34999 } health_report { overall_health: UNKNOWN } } }
I20260927 14:52:02.603039 8430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:52:02.599497 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56600 (local address 127.0.98.1:46477)
0927 14:52:02.599612 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0927 14:52:02.599617 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 14:52:02.599631 (+ 14us) server_negotiation.cc:407] Connection header received
0927 14:52:02.599743 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:52:02.599746 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:52:02.599805 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:52:02.599911 (+ 106us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 14:52:02.600155 (+ 244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:52:02.600641 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:52:02.601570 (+ 929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:52:02.601806 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:52:02.601930 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:52:02.602217 (+ 287us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 14:52:02.602663 (+ 446us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 14:52:02.602801 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0927 14:52:02.602842 (+ 41us) server_negotiation.cc:299] Negotiation successful
0927 14:52:02.602894 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260927 14:52:02.608525 8358 raft_consensus.cc:1261] T da8422bca7c54af4af049dc8c472fc8a P fd36717bc96647548d91ea87dd09dae4 [term 1 FOLLOWER]: Refusing update from remote peer 7a267c5c521e4912aad1c6124b1a92b6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 14:52:02.608525 8218 raft_consensus.cc:1261] T da8422bca7c54af4af049dc8c472fc8a P b9684a38ff254c1287b71a241d319172 [term 1 FOLLOWER]: Refusing update from remote peer 7a267c5c521e4912aad1c6124b1a92b6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 14:52:02.609194 8438 consensus_queue.cc:1035] T da8422bca7c54af4af049dc8c472fc8a P 7a267c5c521e4912aad1c6124b1a92b6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fd36717bc96647548d91ea87dd09dae4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43169 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 14:52:02.609395 8443 consensus_queue.cc:1035] T da8422bca7c54af4af049dc8c472fc8a P 7a267c5c521e4912aad1c6124b1a92b6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9684a38ff254c1287b71a241d319172" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34999 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 14:52:02.616660 8446 mvcc.cc:205] Tried to move back new op lower bound from 7333972879797760000 to 7333972879675633664. Current Snapshot: MvccSnapshot[applied={T|T < 7333972879797760000}]
I20260927 14:52:02.620255 8447 mvcc.cc:205] Tried to move back new op lower bound from 7333972879797760000 to 7333972879675633664. Current Snapshot: MvccSnapshot[applied={T|T < 7333972879797760000 or (T in {7333972879797760000})}]
I20260927 14:52:02.624688 7921 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39078:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 14:52:02.624859 7921 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39078:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 14:52:02.627238 7921 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d4bb90ee61a346e783b1407c8f06dc21: Sending DeleteTablet for 3 replicas of tablet da8422bca7c54af4af049dc8c472fc8a
I20260927 14:52:02.627924 8338 tablet_service.cc:1514] Processing DeleteTablet for tablet da8422bca7c54af4af049dc8c472fc8a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 14:52:02 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:46842
I20260927 14:52:02.628412 8455 tablet_replica.cc:330] T da8422bca7c54af4af049dc8c472fc8a P fd36717bc96647548d91ea87dd09dae4: stopping tablet replica
I20260927 14:52:02.628433 8058 tablet_service.cc:1514] Processing DeleteTablet for tablet da8422bca7c54af4af049dc8c472fc8a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 14:52:02 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:56588
I20260927 14:52:02.628693 8455 raft_consensus.cc:2229] T da8422bca7c54af4af049dc8c472fc8a P fd36717bc96647548d91ea87dd09dae4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 14:52:02.628911 8456 tablet_replica.cc:330] T da8422bca7c54af4af049dc8c472fc8a P 7a267c5c521e4912aad1c6124b1a92b6: stopping tablet replica
I20260927 14:52:02.629012 8455 raft_consensus.cc:2258] T da8422bca7c54af4af049dc8c472fc8a P fd36717bc96647548d91ea87dd09dae4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 14:52:02.628983 8198 tablet_service.cc:1514] Processing DeleteTablet for tablet da8422bca7c54af4af049dc8c472fc8a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 14:52:02 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:52474
I20260927 14:52:02.629335 8456 raft_consensus.cc:2229] T da8422bca7c54af4af049dc8c472fc8a P 7a267c5c521e4912aad1c6124b1a92b6 [term 1 LEADER]: Raft consensus shutting down.
I20260927 14:52:02.629671 8456 raft_consensus.cc:2258] T da8422bca7c54af4af049dc8c472fc8a P 7a267c5c521e4912aad1c6124b1a92b6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 14:52:02.629951 7921 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39078:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 14:52:02.630321 7921 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 14:52:02.630494 8455 ts_tablet_manager.cc:1905] T da8422bca7c54af4af049dc8c472fc8a P fd36717bc96647548d91ea87dd09dae4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 14:52:02.631229 8456 ts_tablet_manager.cc:1905] T da8422bca7c54af4af049dc8c472fc8a P 7a267c5c521e4912aad1c6124b1a92b6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 14:52:02.633285 8455 ts_tablet_manager.cc:1918] T da8422bca7c54af4af049dc8c472fc8a P fd36717bc96647548d91ea87dd09dae4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 14:52:02.633381 8455 log.cc:1196] T da8422bca7c54af4af049dc8c472fc8a P fd36717bc96647548d91ea87dd09dae4: Deleting WAL directory at /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790520684602251-392-0/minicluster-data/ts-2/wal/wals/da8422bca7c54af4af049dc8c472fc8a
I20260927 14:52:02.633687 8455 ts_tablet_manager.cc:1939] T da8422bca7c54af4af049dc8c472fc8a P fd36717bc96647548d91ea87dd09dae4: Deleting consensus metadata
I20260927 14:52:02.634332 8456 ts_tablet_manager.cc:1918] T da8422bca7c54af4af049dc8c472fc8a P 7a267c5c521e4912aad1c6124b1a92b6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 14:52:02.634482 8456 log.cc:1196] T da8422bca7c54af4af049dc8c472fc8a P 7a267c5c521e4912aad1c6124b1a92b6: Deleting WAL directory at /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790520684602251-392-0/minicluster-data/ts-0/wal/wals/da8422bca7c54af4af049dc8c472fc8a
I20260927 14:52:02.634852 8456 ts_tablet_manager.cc:1939] T da8422bca7c54af4af049dc8c472fc8a P 7a267c5c521e4912aad1c6124b1a92b6: Deleting consensus metadata
I20260927 14:52:02.634984 7905 catalog_manager.cc:4841] TS fd36717bc96647548d91ea87dd09dae4 (127.0.98.3:43169): tablet da8422bca7c54af4af049dc8c472fc8a (table test-table [id=d7c8361fa70343b9bc7954252aa9cf0c]) successfully deleted
I20260927 14:52:02.635475 8198 tablet_service.cc:1467] Processing CreateTablet for tablet 58570133cd41478da075e8ccf0e5ebb7 (DEFAULT_TABLE table=test-table [id=30982364c9a3461f9102e519050999c1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 14:52:02.635754 8198 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 58570133cd41478da075e8ccf0e5ebb7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 14:52:02.635972 7905 catalog_manager.cc:4841] TS 7a267c5c521e4912aad1c6124b1a92b6 (127.0.98.1:46477): tablet da8422bca7c54af4af049dc8c472fc8a (table test-table [id=d7c8361fa70343b9bc7954252aa9cf0c]) successfully deleted
I20260927 14:52:02.637311 8058 tablet_service.cc:1467] Processing CreateTablet for tablet 58570133cd41478da075e8ccf0e5ebb7 (DEFAULT_TABLE table=test-table [id=30982364c9a3461f9102e519050999c1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 14:52:02.637549 8058 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 58570133cd41478da075e8ccf0e5ebb7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 14:52:02.638115 8433 tablet_bootstrap.cc:492] T 58570133cd41478da075e8ccf0e5ebb7 P b9684a38ff254c1287b71a241d319172: Bootstrap starting.
I20260927 14:52:02.639055 8433 tablet_bootstrap.cc:654] T 58570133cd41478da075e8ccf0e5ebb7 P b9684a38ff254c1287b71a241d319172: Neither blocks nor log segments found. Creating new log.
I20260927 14:52:02.639703 8432 tablet_bootstrap.cc:492] T 58570133cd41478da075e8ccf0e5ebb7 P 7a267c5c521e4912aad1c6124b1a92b6: Bootstrap starting.
I20260927 14:52:02.639930 8338 tablet_service.cc:1467] Processing CreateTablet for tablet 58570133cd41478da075e8ccf0e5ebb7 (DEFAULT_TABLE table=test-table [id=30982364c9a3461f9102e519050999c1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 14:52:02.640193 8338 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 58570133cd41478da075e8ccf0e5ebb7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 14:52:02.640585 8432 tablet_bootstrap.cc:654] T 58570133cd41478da075e8ccf0e5ebb7 P 7a267c5c521e4912aad1c6124b1a92b6: Neither blocks nor log segments found. Creating new log.
I20260927 14:52:02.641754 8432 tablet_bootstrap.cc:492] T 58570133cd41478da075e8ccf0e5ebb7 P 7a267c5c521e4912aad1c6124b1a92b6: No bootstrap required, opened a new log
I20260927 14:52:02.641768 8433 tablet_bootstrap.cc:492] T 58570133cd41478da075e8ccf0e5ebb7 P b9684a38ff254c1287b71a241d319172: No bootstrap required, opened a new log
I20260927 14:52:02.641832 8432 ts_tablet_manager.cc:1397] T 58570133cd41478da075e8ccf0e5ebb7 P 7a267c5c521e4912aad1c6124b1a92b6: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 14:52:02.641839 8433 ts_tablet_manager.cc:1397] T 58570133cd41478da075e8ccf0e5ebb7 P b9684a38ff254c1287b71a241d319172: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.001s
I20260927 14:52:02.642252 8433 raft_consensus.cc:348] T 58570133cd41478da075e8ccf0e5ebb7 P b9684a38ff254c1287b71a241d319172 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a267c5c521e4912aad1c6124b1a92b6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46477 } } peers { permanent_uuid: "b9684a38ff254c1287b71a241d319172" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34999 } } peers { permanent_uuid: "fd36717bc96647548d91ea87dd09dae4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43169 } }
I20260927 14:52:02.642270 8432 raft_consensus.cc:348] T 58570133cd41478da075e8ccf0e5ebb7 P 7a267c5c521e4912aad1c6124b1a92b6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a267c5c521e4912aad1c6124b1a92b6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46477 } } peers { permanent_uuid: "b9684a38ff254c1287b71a241d319172" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34999 } } peers { permanent_uuid: "fd36717bc96647548d91ea87dd09dae4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43169 } }
I20260927 14:52:02.642354 8433 raft_consensus.cc:374] T 58570133cd41478da075e8ccf0e5ebb7 P b9684a38ff254c1287b71a241d319172 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 14:52:02.642375 8432 raft_consensus.cc:374] T 58570133cd41478da075e8ccf0e5ebb7 P 7a267c5c521e4912aad1c6124b1a92b6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 14:52:02.642385 8433 raft_consensus.cc:729] T 58570133cd41478da075e8ccf0e5ebb7 P b9684a38ff254c1287b71a241d319172 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b9684a38ff254c1287b71a241d319172, State: Initialized, Role: FOLLOWER
I20260927 14:52:02.642406 8432 raft_consensus.cc:729] T 58570133cd41478da075e8ccf0e5ebb7 P 7a267c5c521e4912aad1c6124b1a92b6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a267c5c521e4912aad1c6124b1a92b6, State: Initialized, Role: FOLLOWER
I20260927 14:52:02.642513 8433 consensus_queue.cc:260] T 58570133cd41478da075e8ccf0e5ebb7 P b9684a38ff254c1287b71a241d319172 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a267c5c521e4912aad1c6124b1a92b6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46477 } } peers { permanent_uuid: "b9684a38ff254c1287b71a241d319172" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34999 } } peers { permanent_uuid: "fd36717bc96647548d91ea87dd09dae4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43169 } }
I20260927 14:52:02.642525 8432 consensus_queue.cc:260] T 58570133cd41478da075e8ccf0e5ebb7 P 7a267c5c521e4912aad1c6124b1a92b6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a267c5c521e4912aad1c6124b1a92b6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46477 } } peers { permanent_uuid: "b9684a38ff254c1287b71a241d319172" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34999 } } peers { permanent_uuid: "fd36717bc96647548d91ea87dd09dae4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43169 } }
I20260927 14:52:02.642760 8433 ts_tablet_manager.cc:1428] T 58570133cd41478da075e8ccf0e5ebb7 P b9684a38ff254c1287b71a241d319172: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260927 14:52:02.642783 8432 ts_tablet_manager.cc:1428] T 58570133cd41478da075e8ccf0e5ebb7 P 7a267c5c521e4912aad1c6124b1a92b6: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260927 14:52:02.643056 8434 tablet_bootstrap.cc:492] T 58570133cd41478da075e8ccf0e5ebb7 P fd36717bc96647548d91ea87dd09dae4: Bootstrap starting.
I20260927 14:52:02.643266 8457 tablet_replica.cc:330] T da8422bca7c54af4af049dc8c472fc8a P b9684a38ff254c1287b71a241d319172: stopping tablet replica
I20260927 14:52:02.643965 8457 raft_consensus.cc:2229] T da8422bca7c54af4af049dc8c472fc8a P b9684a38ff254c1287b71a241d319172 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 14:52:02.644021 8434 tablet_bootstrap.cc:654] T 58570133cd41478da075e8ccf0e5ebb7 P fd36717bc96647548d91ea87dd09dae4: Neither blocks nor log segments found. Creating new log.
I20260927 14:52:02.644302 8457 raft_consensus.cc:2258] T da8422bca7c54af4af049dc8c472fc8a P b9684a38ff254c1287b71a241d319172 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 14:52:02.645267 8434 tablet_bootstrap.cc:492] T 58570133cd41478da075e8ccf0e5ebb7 P fd36717bc96647548d91ea87dd09dae4: No bootstrap required, opened a new log
I20260927 14:52:02.645366 8434 ts_tablet_manager.cc:1397] T 58570133cd41478da075e8ccf0e5ebb7 P fd36717bc96647548d91ea87dd09dae4: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.001s
I20260927 14:52:02.645835 8434 raft_consensus.cc:348] T 58570133cd41478da075e8ccf0e5ebb7 P fd36717bc96647548d91ea87dd09dae4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a267c5c521e4912aad1c6124b1a92b6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46477 } } peers { permanent_uuid: "b9684a38ff254c1287b71a241d319172" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34999 } } peers { permanent_uuid: "fd36717bc96647548d91ea87dd09dae4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43169 } }
I20260927 14:52:02.645936 8434 raft_consensus.cc:374] T 58570133cd41478da075e8ccf0e5ebb7 P fd36717bc96647548d91ea87dd09dae4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 14:52:02.645967 8434 raft_consensus.cc:729] T 58570133cd41478da075e8ccf0e5ebb7 P fd36717bc96647548d91ea87dd09dae4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd36717bc96647548d91ea87dd09dae4, State: Initialized, Role: FOLLOWER
I20260927 14:52:02.646102 8434 consensus_queue.cc:260] T 58570133cd41478da075e8ccf0e5ebb7 P fd36717bc96647548d91ea87dd09dae4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a267c5c521e4912aad1c6124b1a92b6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46477 } } peers { permanent_uuid: "b9684a38ff254c1287b71a241d319172" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34999 } } peers { permanent_uuid: "fd36717bc96647548d91ea87dd09dae4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43169 } }
I20260927 14:52:02.646363 8434 ts_tablet_manager.cc:1428] T 58570133cd41478da075e8ccf0e5ebb7 P fd36717bc96647548d91ea87dd09dae4: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260927 14:52:02.647042 8457 ts_tablet_manager.cc:1905] T da8422bca7c54af4af049dc8c472fc8a P b9684a38ff254c1287b71a241d319172: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 14:52:02.649782 8457 ts_tablet_manager.cc:1918] T da8422bca7c54af4af049dc8c472fc8a P b9684a38ff254c1287b71a241d319172: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 14:52:02.649883 8457 log.cc:1196] T da8422bca7c54af4af049dc8c472fc8a P b9684a38ff254c1287b71a241d319172: Deleting WAL directory at /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790520684602251-392-0/minicluster-data/ts-1/wal/wals/da8422bca7c54af4af049dc8c472fc8a
I20260927 14:52:02.650156 8457 ts_tablet_manager.cc:1939] T da8422bca7c54af4af049dc8c472fc8a P b9684a38ff254c1287b71a241d319172: Deleting consensus metadata
I20260927 14:52:02.651309 7908 catalog_manager.cc:4841] TS b9684a38ff254c1287b71a241d319172 (127.0.98.2:34999): tablet da8422bca7c54af4af049dc8c472fc8a (table test-table [id=d7c8361fa70343b9bc7954252aa9cf0c]) successfully deleted
I20260927 14:52:02.767220 8443 raft_consensus.cc:482] T 58570133cd41478da075e8ccf0e5ebb7 P 7a267c5c521e4912aad1c6124b1a92b6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 14:52:02.767371 8443 raft_consensus.cc:504] T 58570133cd41478da075e8ccf0e5ebb7 P 7a267c5c521e4912aad1c6124b1a92b6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a267c5c521e4912aad1c6124b1a92b6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46477 } } peers { permanent_uuid: "b9684a38ff254c1287b71a241d319172" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34999 } } peers { permanent_uuid: "fd36717bc96647548d91ea87dd09dae4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43169 } }
I20260927 14:52:02.767819 8443 leader_election.cc:288] T 58570133cd41478da075e8ccf0e5ebb7 P 7a267c5c521e4912aad1c6124b1a92b6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b9684a38ff254c1287b71a241d319172 (127.0.98.2:34999), fd36717bc96647548d91ea87dd09dae4 (127.0.98.3:43169)
I20260927 14:52:02.768205 8218 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58570133cd41478da075e8ccf0e5ebb7" candidate_uuid: "7a267c5c521e4912aad1c6124b1a92b6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9684a38ff254c1287b71a241d319172" is_pre_election: true
I20260927 14:52:02.768208 8358 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58570133cd41478da075e8ccf0e5ebb7" candidate_uuid: "7a267c5c521e4912aad1c6124b1a92b6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd36717bc96647548d91ea87dd09dae4" is_pre_election: true
I20260927 14:52:02.768359 8218 raft_consensus.cc:2454] T 58570133cd41478da075e8ccf0e5ebb7 P b9684a38ff254c1287b71a241d319172 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a267c5c521e4912aad1c6124b1a92b6 in term 0.
I20260927 14:52:02.768370 8358 raft_consensus.cc:2454] T 58570133cd41478da075e8ccf0e5ebb7 P fd36717bc96647548d91ea87dd09dae4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a267c5c521e4912aad1c6124b1a92b6 in term 0.
I20260927 14:52:02.768646 8012 leader_election.cc:302] T 58570133cd41478da075e8ccf0e5ebb7 P 7a267c5c521e4912aad1c6124b1a92b6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a267c5c521e4912aad1c6124b1a92b6, b9684a38ff254c1287b71a241d319172; no voters:
I20260927 14:52:02.768798 8443 raft_consensus.cc:2781] T 58570133cd41478da075e8ccf0e5ebb7 P 7a267c5c521e4912aad1c6124b1a92b6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 14:52:02.768891 8443 raft_consensus.cc:482] T 58570133cd41478da075e8ccf0e5ebb7 P 7a267c5c521e4912aad1c6124b1a92b6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 14:52:02.768921 8443 raft_consensus.cc:3037] T 58570133cd41478da075e8ccf0e5ebb7 P 7a267c5c521e4912aad1c6124b1a92b6 [term 0 FOLLOWER]: Advancing to term 1
I20260927 14:52:02.769896 8443 raft_consensus.cc:504] T 58570133cd41478da075e8ccf0e5ebb7 P 7a267c5c521e4912aad1c6124b1a92b6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a267c5c521e4912aad1c6124b1a92b6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46477 } } peers { permanent_uuid: "b9684a38ff254c1287b71a241d319172" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34999 } } peers { permanent_uuid: "fd36717bc96647548d91ea87dd09dae4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43169 } }
I20260927 14:52:02.770263 8443 leader_election.cc:288] T 58570133cd41478da075e8ccf0e5ebb7 P 7a267c5c521e4912aad1c6124b1a92b6 [CANDIDATE]: Term 1 election: Requested vote from peers b9684a38ff254c1287b71a241d319172 (127.0.98.2:34999), fd36717bc96647548d91ea87dd09dae4 (127.0.98.3:43169)
I20260927 14:52:02.770630 8358 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58570133cd41478da075e8ccf0e5ebb7" candidate_uuid: "7a267c5c521e4912aad1c6124b1a92b6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd36717bc96647548d91ea87dd09dae4"
I20260927 14:52:02.770663 8218 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58570133cd41478da075e8ccf0e5ebb7" candidate_uuid: "7a267c5c521e4912aad1c6124b1a92b6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9684a38ff254c1287b71a241d319172"
I20260927 14:52:02.770756 8358 raft_consensus.cc:3037] T 58570133cd41478da075e8ccf0e5ebb7 P fd36717bc96647548d91ea87dd09dae4 [term 0 FOLLOWER]: Advancing to term 1
I20260927 14:52:02.770848 8218 raft_consensus.cc:3037] T 58570133cd41478da075e8ccf0e5ebb7 P b9684a38ff254c1287b71a241d319172 [term 0 FOLLOWER]: Advancing to term 1
I20260927 14:52:02.771669 8358 raft_consensus.cc:2454] T 58570133cd41478da075e8ccf0e5ebb7 P fd36717bc96647548d91ea87dd09dae4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a267c5c521e4912aad1c6124b1a92b6 in term 1.
I20260927 14:52:02.771971 8009 leader_election.cc:302] T 58570133cd41478da075e8ccf0e5ebb7 P 7a267c5c521e4912aad1c6124b1a92b6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a267c5c521e4912aad1c6124b1a92b6, fd36717bc96647548d91ea87dd09dae4; no voters:
I20260927 14:52:02.772042 8218 raft_consensus.cc:2454] T 58570133cd41478da075e8ccf0e5ebb7 P b9684a38ff254c1287b71a241d319172 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a267c5c521e4912aad1c6124b1a92b6 in term 1.
I20260927 14:52:02.772157 8443 raft_consensus.cc:2781] T 58570133cd41478da075e8ccf0e5ebb7 P 7a267c5c521e4912aad1c6124b1a92b6 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 14:52:02.772238 8443 raft_consensus.cc:686] T 58570133cd41478da075e8ccf0e5ebb7 P 7a267c5c521e4912aad1c6124b1a92b6 [term 1 LEADER]: Becoming Leader. State: Replica: 7a267c5c521e4912aad1c6124b1a92b6, State: Running, Role: LEADER
I20260927 14:52:02.772382 8443 consensus_queue.cc:237] T 58570133cd41478da075e8ccf0e5ebb7 P 7a267c5c521e4912aad1c6124b1a92b6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a267c5c521e4912aad1c6124b1a92b6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46477 } } peers { permanent_uuid: "b9684a38ff254c1287b71a241d319172" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34999 } } peers { permanent_uuid: "fd36717bc96647548d91ea87dd09dae4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43169 } }
I20260927 14:52:02.774210 7921 catalog_manager.cc:5495] T 58570133cd41478da075e8ccf0e5ebb7 P 7a267c5c521e4912aad1c6124b1a92b6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7a267c5c521e4912aad1c6124b1a92b6 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "7a267c5c521e4912aad1c6124b1a92b6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a267c5c521e4912aad1c6124b1a92b6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46477 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b9684a38ff254c1287b71a241d319172" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34999 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fd36717bc96647548d91ea87dd09dae4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43169 } health_report { overall_health: UNKNOWN } } }
Sep 27 14:52:02 dist-test-slave-xk1g krb5kdc[7876](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790520721, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260927 14:52:02.819322 8467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:52:02.810707 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42843 (local address 127.0.0.1:39084)
0927 14:52:02.811096 (+ 389us) negotiation.cc:107] Waiting for socket to connect
0927 14:52:02.811112 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 14:52:02.811264 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:52:02.811522 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:52:02.811525 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:52:02.811959 (+ 434us) client_negotiation.cc:189] Negotiated authn=SASL
0927 14:52:02.812053 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:52:02.812060 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:52:02.812859 (+ 799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:52:02.812862 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:52:02.813540 (+ 678us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:52:02.813550 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:52:02.814019 (+ 469us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:52:02.814022 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:52:02.814094 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:52:02.814858 (+ 764us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 14:52:02.814869 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 14:52:02.816213 (+ 1344us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 14:52:02.817737 (+ 1524us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 14:52:02.817748 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 14:52:02.817755 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0927 14:52:02.817988 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 14:52:02.818257 (+ 269us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 14:52:02.818261 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 14:52:02.818263 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 14:52:02.818344 (+ 81us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 14:52:02.818718 (+ 374us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 14:52:02.818727 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 14:52:02.818841 (+ 114us) client_negotiation.cc:769] Sending connection context
0927 14:52:02.818959 (+ 118us) client_negotiation.cc:240] Negotiation successful
0927 14:52:02.819112 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":282,"thread_start_us":100,"threads_started":1}
I20260927 14:52:02.819322 8468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:52:02.810803 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39084 (local address 127.0.98.62:42843)
0927 14:52:02.811077 (+ 274us) server_negotiation.cc:206] Beginning negotiation
0927 14:52:02.811085 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 14:52:02.811202 (+ 117us) server_negotiation.cc:407] Connection header received
0927 14:52:02.811365 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:52:02.811368 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:52:02.811414 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:52:02.811507 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:52:02.812177 (+ 670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:52:02.812709 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:52:02.813651 (+ 942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:52:02.813912 (+ 261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:52:02.814020 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:52:02.816349 (+ 2329us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:52:02.816353 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:52:02.816356 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 14:52:02.816380 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:52:02.817633 (+ 1253us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:52:02.818086 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:52:02.818092 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:52:02.818094 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:52:02.818169 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:52:02.818438 (+ 269us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:52:02.818441 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:52:02.818442 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:52:02.818616 (+ 174us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:52:02.818719 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0927 14:52:02.819004 (+ 285us) server_negotiation.cc:299] Negotiation successful
0927 14:52:02.819172 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":68,"server-negotiator.queue_time_us":184,"thread_start_us":94,"threads_started":1}
I20260927 14:52:02.823300 7921 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:39084:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260927 14:52:02.824092 7921 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 14:52:02.828927 8058 tablet_service.cc:1467] Processing CreateTablet for tablet db1c8c3872024b1bafb462ad9637a6c3 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=a6fc643c1c984b13a48570387b5b8649]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 14:52:02.829021 8198 tablet_service.cc:1467] Processing CreateTablet for tablet db1c8c3872024b1bafb462ad9637a6c3 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=a6fc643c1c984b13a48570387b5b8649]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 14:52:02.829416 8058 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet db1c8c3872024b1bafb462ad9637a6c3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 14:52:02.829437 8198 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet db1c8c3872024b1bafb462ad9637a6c3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 14:52:02.831902 8432 tablet_bootstrap.cc:492] T db1c8c3872024b1bafb462ad9637a6c3 P 7a267c5c521e4912aad1c6124b1a92b6: Bootstrap starting.
I20260927 14:52:02.832115 8433 tablet_bootstrap.cc:492] T db1c8c3872024b1bafb462ad9637a6c3 P b9684a38ff254c1287b71a241d319172: Bootstrap starting.
I20260927 14:52:02.832204 8338 tablet_service.cc:1467] Processing CreateTablet for tablet db1c8c3872024b1bafb462ad9637a6c3 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=a6fc643c1c984b13a48570387b5b8649]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 14:52:02.832564 8338 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet db1c8c3872024b1bafb462ad9637a6c3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 14:52:02.832769 8432 tablet_bootstrap.cc:654] T db1c8c3872024b1bafb462ad9637a6c3 P 7a267c5c521e4912aad1c6124b1a92b6: Neither blocks nor log segments found. Creating new log.
I20260927 14:52:02.833154 8433 tablet_bootstrap.cc:654] T db1c8c3872024b1bafb462ad9637a6c3 P b9684a38ff254c1287b71a241d319172: Neither blocks nor log segments found. Creating new log.
I20260927 14:52:02.834018 8432 tablet_bootstrap.cc:492] T db1c8c3872024b1bafb462ad9637a6c3 P 7a267c5c521e4912aad1c6124b1a92b6: No bootstrap required, opened a new log
I20260927 14:52:02.834105 8432 ts_tablet_manager.cc:1397] T db1c8c3872024b1bafb462ad9637a6c3 P 7a267c5c521e4912aad1c6124b1a92b6: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260927 14:52:02.834301 8433 tablet_bootstrap.cc:492] T db1c8c3872024b1bafb462ad9637a6c3 P b9684a38ff254c1287b71a241d319172: No bootstrap required, opened a new log
I20260927 14:52:02.834367 8433 ts_tablet_manager.cc:1397] T db1c8c3872024b1bafb462ad9637a6c3 P b9684a38ff254c1287b71a241d319172: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260927 14:52:02.834519 8432 raft_consensus.cc:348] T db1c8c3872024b1bafb462ad9637a6c3 P 7a267c5c521e4912aad1c6124b1a92b6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a267c5c521e4912aad1c6124b1a92b6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46477 } } peers { permanent_uuid: "b9684a38ff254c1287b71a241d319172" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34999 } } peers { permanent_uuid: "fd36717bc96647548d91ea87dd09dae4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43169 } }
I20260927 14:52:02.834623 8432 raft_consensus.cc:374] T db1c8c3872024b1bafb462ad9637a6c3 P 7a267c5c521e4912aad1c6124b1a92b6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 14:52:02.834656 8432 raft_consensus.cc:729] T db1c8c3872024b1bafb462ad9637a6c3 P 7a267c5c521e4912aad1c6124b1a92b6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a267c5c521e4912aad1c6124b1a92b6, State: Initialized, Role: FOLLOWER
I20260927 14:52:02.834779 8432 consensus_queue.cc:260] T db1c8c3872024b1bafb462ad9637a6c3 P 7a267c5c521e4912aad1c6124b1a92b6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a267c5c521e4912aad1c6124b1a92b6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46477 } } peers { permanent_uuid: "b9684a38ff254c1287b71a241d319172" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34999 } } peers { permanent_uuid: "fd36717bc96647548d91ea87dd09dae4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43169 } }
I20260927 14:52:02.834785 8433 raft_consensus.cc:348] T db1c8c3872024b1bafb462ad9637a6c3 P b9684a38ff254c1287b71a241d319172 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a267c5c521e4912aad1c6124b1a92b6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46477 } } peers { permanent_uuid: "b9684a38ff254c1287b71a241d319172" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34999 } } peers { permanent_uuid: "fd36717bc96647548d91ea87dd09dae4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43169 } }
I20260927 14:52:02.834872 8433 raft_consensus.cc:374] T db1c8c3872024b1bafb462ad9637a6c3 P b9684a38ff254c1287b71a241d319172 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 14:52:02.834905 8433 raft_consensus.cc:729] T db1c8c3872024b1bafb462ad9637a6c3 P b9684a38ff254c1287b71a241d319172 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b9684a38ff254c1287b71a241d319172, State: Initialized, Role: FOLLOWER
I20260927 14:52:02.835067 8432 ts_tablet_manager.cc:1428] T db1c8c3872024b1bafb462ad9637a6c3 P 7a267c5c521e4912aad1c6124b1a92b6: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260927 14:52:02.835021 8433 consensus_queue.cc:260] T db1c8c3872024b1bafb462ad9637a6c3 P b9684a38ff254c1287b71a241d319172 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a267c5c521e4912aad1c6124b1a92b6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46477 } } peers { permanent_uuid: "b9684a38ff254c1287b71a241d319172" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34999 } } peers { permanent_uuid: "fd36717bc96647548d91ea87dd09dae4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43169 } }
I20260927 14:52:02.835172 8434 tablet_bootstrap.cc:492] T db1c8c3872024b1bafb462ad9637a6c3 P fd36717bc96647548d91ea87dd09dae4: Bootstrap starting.
I20260927 14:52:02.835278 8433 ts_tablet_manager.cc:1428] T db1c8c3872024b1bafb462ad9637a6c3 P b9684a38ff254c1287b71a241d319172: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260927 14:52:02.835246 8438 tablet_replica.cc:439] T db1c8c3872024b1bafb462ad9637a6c3 P 7a267c5c521e4912aad1c6124b1a92b6: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a267c5c521e4912aad1c6124b1a92b6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46477 } } peers { permanent_uuid: "b9684a38ff254c1287b71a241d319172" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34999 } } peers { permanent_uuid: "fd36717bc96647548d91ea87dd09dae4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43169 } } }
I20260927 14:52:02.835357 8438 tablet_replica.cc:442] T db1c8c3872024b1bafb462ad9637a6c3 P 7a267c5c521e4912aad1c6124b1a92b6: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 14:52:02.835798 8439 tablet_replica.cc:439] T db1c8c3872024b1bafb462ad9637a6c3 P b9684a38ff254c1287b71a241d319172: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a267c5c521e4912aad1c6124b1a92b6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46477 } } peers { permanent_uuid: "b9684a38ff254c1287b71a241d319172" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34999 } } peers { permanent_uuid: "fd36717bc96647548d91ea87dd09dae4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43169 } } }
I20260927 14:52:02.835937 8439 tablet_replica.cc:442] T db1c8c3872024b1bafb462ad9637a6c3 P b9684a38ff254c1287b71a241d319172: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 14:52:02.836017 8434 tablet_bootstrap.cc:654] T db1c8c3872024b1bafb462ad9637a6c3 P fd36717bc96647548d91ea87dd09dae4: Neither blocks nor log segments found. Creating new log.
I20260927 14:52:02.837499 8434 tablet_bootstrap.cc:492] T db1c8c3872024b1bafb462ad9637a6c3 P fd36717bc96647548d91ea87dd09dae4: No bootstrap required, opened a new log
I20260927 14:52:02.837584 8434 ts_tablet_manager.cc:1397] T db1c8c3872024b1bafb462ad9637a6c3 P fd36717bc96647548d91ea87dd09dae4: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260927 14:52:02.837996 8434 raft_consensus.cc:348] T db1c8c3872024b1bafb462ad9637a6c3 P fd36717bc96647548d91ea87dd09dae4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a267c5c521e4912aad1c6124b1a92b6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46477 } } peers { permanent_uuid: "b9684a38ff254c1287b71a241d319172" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34999 } } peers { permanent_uuid: "fd36717bc96647548d91ea87dd09dae4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43169 } }
I20260927 14:52:02.838111 8434 raft_consensus.cc:374] T db1c8c3872024b1bafb462ad9637a6c3 P fd36717bc96647548d91ea87dd09dae4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 14:52:02.838151 8434 raft_consensus.cc:729] T db1c8c3872024b1bafb462ad9637a6c3 P fd36717bc96647548d91ea87dd09dae4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd36717bc96647548d91ea87dd09dae4, State: Initialized, Role: FOLLOWER
I20260927 14:52:02.838284 8434 consensus_queue.cc:260] T db1c8c3872024b1bafb462ad9637a6c3 P fd36717bc96647548d91ea87dd09dae4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a267c5c521e4912aad1c6124b1a92b6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46477 } } peers { permanent_uuid: "b9684a38ff254c1287b71a241d319172" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34999 } } peers { permanent_uuid: "fd36717bc96647548d91ea87dd09dae4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43169 } }
I20260927 14:52:02.838560 8434 ts_tablet_manager.cc:1428] T db1c8c3872024b1bafb462ad9637a6c3 P fd36717bc96647548d91ea87dd09dae4: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260927 14:52:02.838507 8440 tablet_replica.cc:439] T db1c8c3872024b1bafb462ad9637a6c3 P fd36717bc96647548d91ea87dd09dae4: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a267c5c521e4912aad1c6124b1a92b6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46477 } } peers { permanent_uuid: "b9684a38ff254c1287b71a241d319172" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34999 } } peers { permanent_uuid: "fd36717bc96647548d91ea87dd09dae4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43169 } } }
I20260927 14:52:02.838641 8440 tablet_replica.cc:442] T db1c8c3872024b1bafb462ad9637a6c3 P fd36717bc96647548d91ea87dd09dae4: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 14:52:02.899094 8438 raft_consensus.cc:482] T db1c8c3872024b1bafb462ad9637a6c3 P 7a267c5c521e4912aad1c6124b1a92b6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 14:52:02.899221 8438 raft_consensus.cc:504] T db1c8c3872024b1bafb462ad9637a6c3 P 7a267c5c521e4912aad1c6124b1a92b6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a267c5c521e4912aad1c6124b1a92b6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46477 } } peers { permanent_uuid: "b9684a38ff254c1287b71a241d319172" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34999 } } peers { permanent_uuid: "fd36717bc96647548d91ea87dd09dae4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43169 } }
I20260927 14:52:02.899551 8438 leader_election.cc:288] T db1c8c3872024b1bafb462ad9637a6c3 P 7a267c5c521e4912aad1c6124b1a92b6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b9684a38ff254c1287b71a241d319172 (127.0.98.2:34999), fd36717bc96647548d91ea87dd09dae4 (127.0.98.3:43169)
I20260927 14:52:02.899981 8218 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db1c8c3872024b1bafb462ad9637a6c3" candidate_uuid: "7a267c5c521e4912aad1c6124b1a92b6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9684a38ff254c1287b71a241d319172" is_pre_election: true
I20260927 14:52:02.899986 8358 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db1c8c3872024b1bafb462ad9637a6c3" candidate_uuid: "7a267c5c521e4912aad1c6124b1a92b6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd36717bc96647548d91ea87dd09dae4" is_pre_election: true
I20260927 14:52:02.900110 8218 raft_consensus.cc:2454] T db1c8c3872024b1bafb462ad9637a6c3 P b9684a38ff254c1287b71a241d319172 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a267c5c521e4912aad1c6124b1a92b6 in term 0.
I20260927 14:52:02.900157 8358 raft_consensus.cc:2454] T db1c8c3872024b1bafb462ad9637a6c3 P fd36717bc96647548d91ea87dd09dae4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a267c5c521e4912aad1c6124b1a92b6 in term 0.
I20260927 14:52:02.900489 8009 leader_election.cc:302] T db1c8c3872024b1bafb462ad9637a6c3 P 7a267c5c521e4912aad1c6124b1a92b6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 7a267c5c521e4912aad1c6124b1a92b6, b9684a38ff254c1287b71a241d319172, fd36717bc96647548d91ea87dd09dae4; no voters:
I20260927 14:52:02.900682 8438 raft_consensus.cc:2781] T db1c8c3872024b1bafb462ad9637a6c3 P 7a267c5c521e4912aad1c6124b1a92b6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 14:52:02.900753 8438 raft_consensus.cc:482] T db1c8c3872024b1bafb462ad9637a6c3 P 7a267c5c521e4912aad1c6124b1a92b6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 14:52:02.900805 8438 raft_consensus.cc:3037] T db1c8c3872024b1bafb462ad9637a6c3 P 7a267c5c521e4912aad1c6124b1a92b6 [term 0 FOLLOWER]: Advancing to term 1
I20260927 14:52:02.901840 8438 raft_consensus.cc:504] T db1c8c3872024b1bafb462ad9637a6c3 P 7a267c5c521e4912aad1c6124b1a92b6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a267c5c521e4912aad1c6124b1a92b6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46477 } } peers { permanent_uuid: "b9684a38ff254c1287b71a241d319172" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34999 } } peers { permanent_uuid: "fd36717bc96647548d91ea87dd09dae4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43169 } }
I20260927 14:52:02.902174 8438 leader_election.cc:288] T db1c8c3872024b1bafb462ad9637a6c3 P 7a267c5c521e4912aad1c6124b1a92b6 [CANDIDATE]: Term 1 election: Requested vote from peers b9684a38ff254c1287b71a241d319172 (127.0.98.2:34999), fd36717bc96647548d91ea87dd09dae4 (127.0.98.3:43169)
I20260927 14:52:02.902554 8218 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db1c8c3872024b1bafb462ad9637a6c3" candidate_uuid: "7a267c5c521e4912aad1c6124b1a92b6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9684a38ff254c1287b71a241d319172"
I20260927 14:52:02.902699 8218 raft_consensus.cc:3037] T db1c8c3872024b1bafb462ad9637a6c3 P b9684a38ff254c1287b71a241d319172 [term 0 FOLLOWER]: Advancing to term 1
I20260927 14:52:02.902699 8358 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db1c8c3872024b1bafb462ad9637a6c3" candidate_uuid: "7a267c5c521e4912aad1c6124b1a92b6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd36717bc96647548d91ea87dd09dae4"
I20260927 14:52:02.902833 8358 raft_consensus.cc:3037] T db1c8c3872024b1bafb462ad9637a6c3 P fd36717bc96647548d91ea87dd09dae4 [term 0 FOLLOWER]: Advancing to term 1
I20260927 14:52:02.903648 8218 raft_consensus.cc:2454] T db1c8c3872024b1bafb462ad9637a6c3 P b9684a38ff254c1287b71a241d319172 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a267c5c521e4912aad1c6124b1a92b6 in term 1.
I20260927 14:52:02.903695 8358 raft_consensus.cc:2454] T db1c8c3872024b1bafb462ad9637a6c3 P fd36717bc96647548d91ea87dd09dae4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a267c5c521e4912aad1c6124b1a92b6 in term 1.
I20260927 14:52:02.903971 8012 leader_election.cc:302] T db1c8c3872024b1bafb462ad9637a6c3 P 7a267c5c521e4912aad1c6124b1a92b6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a267c5c521e4912aad1c6124b1a92b6, b9684a38ff254c1287b71a241d319172; no voters:
I20260927 14:52:02.904151 8438 raft_consensus.cc:2781] T db1c8c3872024b1bafb462ad9637a6c3 P 7a267c5c521e4912aad1c6124b1a92b6 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 14:52:02.904287 8438 raft_consensus.cc:686] T db1c8c3872024b1bafb462ad9637a6c3 P 7a267c5c521e4912aad1c6124b1a92b6 [term 1 LEADER]: Becoming Leader. State: Replica: 7a267c5c521e4912aad1c6124b1a92b6, State: Running, Role: LEADER
I20260927 14:52:02.904417 8438 consensus_queue.cc:237] T db1c8c3872024b1bafb462ad9637a6c3 P 7a267c5c521e4912aad1c6124b1a92b6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a267c5c521e4912aad1c6124b1a92b6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46477 } } peers { permanent_uuid: "b9684a38ff254c1287b71a241d319172" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34999 } } peers { permanent_uuid: "fd36717bc96647548d91ea87dd09dae4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43169 } }
I20260927 14:52:02.904994 8443 tablet_replica.cc:439] T db1c8c3872024b1bafb462ad9637a6c3 P 7a267c5c521e4912aad1c6124b1a92b6: TxnStatusTablet state changed. Reason: New leader 7a267c5c521e4912aad1c6124b1a92b6. Latest consensus state: current_term: 1 leader_uuid: "7a267c5c521e4912aad1c6124b1a92b6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a267c5c521e4912aad1c6124b1a92b6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46477 } } peers { permanent_uuid: "b9684a38ff254c1287b71a241d319172" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34999 } } peers { permanent_uuid: "fd36717bc96647548d91ea87dd09dae4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43169 } } }
I20260927 14:52:02.905110 8443 tablet_replica.cc:442] T db1c8c3872024b1bafb462ad9637a6c3 P 7a267c5c521e4912aad1c6124b1a92b6: This TxnStatusTablet replica's current role is: LEADER
I20260927 14:52:02.905486 8473 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260927 14:52:02.905647 8473 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260927 14:52:02.906024 7920 catalog_manager.cc:5495] T db1c8c3872024b1bafb462ad9637a6c3 P 7a267c5c521e4912aad1c6124b1a92b6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7a267c5c521e4912aad1c6124b1a92b6 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "7a267c5c521e4912aad1c6124b1a92b6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a267c5c521e4912aad1c6124b1a92b6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46477 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b9684a38ff254c1287b71a241d319172" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34999 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fd36717bc96647548d91ea87dd09dae4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43169 } health_report { overall_health: UNKNOWN } } }
I20260927 14:52:02.916764 8430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:52:02.913225 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56616 (local address 127.0.98.1:46477)
0927 14:52:02.913341 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0927 14:52:02.913345 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 14:52:02.913445 (+ 100us) server_negotiation.cc:407] Connection header received
0927 14:52:02.913596 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:52:02.913598 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:52:02.913636 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:52:02.913699 (+ 63us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 14:52:02.914009 (+ 310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:52:02.914508 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:52:02.915451 (+ 943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:52:02.915903 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:52:02.916042 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:52:02.916326 (+ 284us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 14:52:02.916405 (+ 79us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 14:52:02.916487 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0927 14:52:02.916589 (+ 102us) server_negotiation.cc:299] Negotiation successful
0927 14:52:02.916637 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260927 14:52:02.916790 8467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:52:02.913135 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:46477 (local address 127.0.0.1:56616)
0927 14:52:02.913368 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0927 14:52:02.913384 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 14:52:02.913497 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:52:02.913748 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:52:02.913751 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:52:02.913761 (+ 10us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 14:52:02.913867 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:52:02.913873 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:52:02.914637 (+ 764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:52:02.914642 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:52:02.915322 (+ 680us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:52:02.915331 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:52:02.916038 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:52:02.916042 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:52:02.916146 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:52:02.916189 (+ 43us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 14:52:02.916502 (+ 313us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 14:52:02.916517 (+ 15us) client_negotiation.cc:769] Sending connection context
0927 14:52:02.916579 (+ 62us) client_negotiation.cc:240] Negotiation successful
0927 14:52:02.916671 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":141}
I20260927 14:52:02.919307 8218 raft_consensus.cc:1261] T db1c8c3872024b1bafb462ad9637a6c3 P b9684a38ff254c1287b71a241d319172 [term 1 FOLLOWER]: Refusing update from remote peer 7a267c5c521e4912aad1c6124b1a92b6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 14:52:02.919375 8358 raft_consensus.cc:1261] T db1c8c3872024b1bafb462ad9637a6c3 P fd36717bc96647548d91ea87dd09dae4 [term 1 FOLLOWER]: Refusing update from remote peer 7a267c5c521e4912aad1c6124b1a92b6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 14:52:02.919757 8438 consensus_queue.cc:1035] T db1c8c3872024b1bafb462ad9637a6c3 P 7a267c5c521e4912aad1c6124b1a92b6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fd36717bc96647548d91ea87dd09dae4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43169 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 14:52:02.919927 8443 consensus_queue.cc:1035] T db1c8c3872024b1bafb462ad9637a6c3 P 7a267c5c521e4912aad1c6124b1a92b6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9684a38ff254c1287b71a241d319172" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34999 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 14:52:02.921316 8439 tablet_replica.cc:439] T db1c8c3872024b1bafb462ad9637a6c3 P b9684a38ff254c1287b71a241d319172: TxnStatusTablet state changed. Reason: New leader 7a267c5c521e4912aad1c6124b1a92b6. Latest consensus state: current_term: 1 leader_uuid: "7a267c5c521e4912aad1c6124b1a92b6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a267c5c521e4912aad1c6124b1a92b6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46477 } } peers { permanent_uuid: "b9684a38ff254c1287b71a241d319172" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34999 } } peers { permanent_uuid: "fd36717bc96647548d91ea87dd09dae4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43169 } } }
I20260927 14:52:02.921424 8439 tablet_replica.cc:442] T db1c8c3872024b1bafb462ad9637a6c3 P b9684a38ff254c1287b71a241d319172: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 14:52:02.922901 8443 tablet_replica.cc:439] T db1c8c3872024b1bafb462ad9637a6c3 P 7a267c5c521e4912aad1c6124b1a92b6: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "7a267c5c521e4912aad1c6124b1a92b6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a267c5c521e4912aad1c6124b1a92b6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46477 } } peers { permanent_uuid: "b9684a38ff254c1287b71a241d319172" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34999 } } peers { permanent_uuid: "fd36717bc96647548d91ea87dd09dae4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43169 } } }
I20260927 14:52:02.922999 8443 tablet_replica.cc:442] T db1c8c3872024b1bafb462ad9637a6c3 P 7a267c5c521e4912aad1c6124b1a92b6: This TxnStatusTablet replica's current role is: LEADER
I20260927 14:52:02.924923 8440 tablet_replica.cc:439] T db1c8c3872024b1bafb462ad9637a6c3 P fd36717bc96647548d91ea87dd09dae4: TxnStatusTablet state changed. Reason: New leader 7a267c5c521e4912aad1c6124b1a92b6. Latest consensus state: current_term: 1 leader_uuid: "7a267c5c521e4912aad1c6124b1a92b6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a267c5c521e4912aad1c6124b1a92b6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46477 } } peers { permanent_uuid: "b9684a38ff254c1287b71a241d319172" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34999 } } peers { permanent_uuid: "fd36717bc96647548d91ea87dd09dae4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43169 } } }
I20260927 14:52:02.925050 8440 tablet_replica.cc:442] T db1c8c3872024b1bafb462ad9637a6c3 P fd36717bc96647548d91ea87dd09dae4: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 14:52:02.925602 8439 tablet_replica.cc:439] T db1c8c3872024b1bafb462ad9637a6c3 P b9684a38ff254c1287b71a241d319172: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "7a267c5c521e4912aad1c6124b1a92b6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a267c5c521e4912aad1c6124b1a92b6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46477 } } peers { permanent_uuid: "b9684a38ff254c1287b71a241d319172" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34999 } } peers { permanent_uuid: "fd36717bc96647548d91ea87dd09dae4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43169 } } }
I20260927 14:52:02.925710 8439 tablet_replica.cc:442] T db1c8c3872024b1bafb462ad9637a6c3 P b9684a38ff254c1287b71a241d319172: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 14:52:02.927974 8438 tablet_replica.cc:439] T db1c8c3872024b1bafb462ad9637a6c3 P 7a267c5c521e4912aad1c6124b1a92b6: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "7a267c5c521e4912aad1c6124b1a92b6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a267c5c521e4912aad1c6124b1a92b6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46477 } } peers { permanent_uuid: "b9684a38ff254c1287b71a241d319172" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34999 } } peers { permanent_uuid: "fd36717bc96647548d91ea87dd09dae4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43169 } } }
I20260927 14:52:02.928077 8438 tablet_replica.cc:442] T db1c8c3872024b1bafb462ad9637a6c3 P 7a267c5c521e4912aad1c6124b1a92b6: This TxnStatusTablet replica's current role is: LEADER
I20260927 14:52:02.928958 8440 tablet_replica.cc:439] T db1c8c3872024b1bafb462ad9637a6c3 P fd36717bc96647548d91ea87dd09dae4: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "7a267c5c521e4912aad1c6124b1a92b6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a267c5c521e4912aad1c6124b1a92b6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46477 } } peers { permanent_uuid: "b9684a38ff254c1287b71a241d319172" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34999 } } peers { permanent_uuid: "fd36717bc96647548d91ea87dd09dae4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43169 } } }
I20260927 14:52:02.929086 8440 tablet_replica.cc:442] T db1c8c3872024b1bafb462ad9637a6c3 P fd36717bc96647548d91ea87dd09dae4: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 14:52:02.940219 8430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:52:02.936820 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:35037 (local address 127.0.98.1:46477)
0927 14:52:02.936949 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0927 14:52:02.936953 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 14:52:02.937136 (+ 183us) server_negotiation.cc:407] Connection header received
0927 14:52:02.937276 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:52:02.937278 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:52:02.937316 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:52:02.937377 (+ 61us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 14:52:02.937604 (+ 227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:52:02.938086 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:52:02.939263 (+ 1177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:52:02.939462 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:52:02.939537 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:52:02.939825 (+ 288us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 14:52:02.939888 (+ 63us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 14:52:02.939947 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0927 14:52:02.940058 (+ 111us) server_negotiation.cc:299] Negotiation successful
0927 14:52:02.940098 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260927 14:52:02.940325 8481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:52:02.936767 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:46477 (local address 127.0.98.1:35037)
0927 14:52:02.937036 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0927 14:52:02.937082 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0927 14:52:02.937186 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:52:02.937407 (+ 221us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:52:02.937410 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:52:02.937418 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 14:52:02.937512 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:52:02.937517 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:52:02.938182 (+ 665us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:52:02.938185 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:52:02.939152 (+ 967us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:52:02.939162 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:52:02.939563 (+ 401us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:52:02.939567 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:52:02.939649 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:52:02.939696 (+ 47us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 14:52:02.939973 (+ 277us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 14:52:02.939979 (+ 6us) client_negotiation.cc:769] Sending connection context
0927 14:52:02.940043 (+ 64us) client_negotiation.cc:240] Negotiation successful
0927 14:52:02.940130 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":172,"mutex_wait_us":3,"thread_start_us":108,"threads_started":1}
I20260927 14:52:02.946798 8358 raft_consensus.cc:1261] T 58570133cd41478da075e8ccf0e5ebb7 P fd36717bc96647548d91ea87dd09dae4 [term 1 FOLLOWER]: Refusing update from remote peer 7a267c5c521e4912aad1c6124b1a92b6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 14:52:02.946898 8218 raft_consensus.cc:1261] T 58570133cd41478da075e8ccf0e5ebb7 P b9684a38ff254c1287b71a241d319172 [term 1 FOLLOWER]: Refusing update from remote peer 7a267c5c521e4912aad1c6124b1a92b6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 14:52:02.947180 8443 consensus_queue.cc:1035] T 58570133cd41478da075e8ccf0e5ebb7 P 7a267c5c521e4912aad1c6124b1a92b6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fd36717bc96647548d91ea87dd09dae4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43169 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 14:52:02.947319 8438 consensus_queue.cc:1035] T 58570133cd41478da075e8ccf0e5ebb7 P 7a267c5c521e4912aad1c6124b1a92b6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9684a38ff254c1287b71a241d319172" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34999 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 14:52:02.955216 8448 mvcc.cc:205] Tried to move back new op lower bound from 7333972881186676736 to 7333972880477057024. Current Snapshot: MvccSnapshot[applied={T|T < 7333972881186676736}]
I20260927 14:52:02.982864 7920 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39078:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 14:52:02.983068 7920 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39078:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 14:52:02.985119 7920 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d4bb90ee61a346e783b1407c8f06dc21: Sending DeleteTablet for 3 replicas of tablet 58570133cd41478da075e8ccf0e5ebb7
I20260927 14:52:02.985703 8198 tablet_service.cc:1514] Processing DeleteTablet for tablet 58570133cd41478da075e8ccf0e5ebb7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 14:52:02 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:52474
I20260927 14:52:02.985721 8058 tablet_service.cc:1514] Processing DeleteTablet for tablet 58570133cd41478da075e8ccf0e5ebb7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 14:52:02 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:56588
I20260927 14:52:02.985894 8457 tablet_replica.cc:330] T 58570133cd41478da075e8ccf0e5ebb7 P b9684a38ff254c1287b71a241d319172: stopping tablet replica
I20260927 14:52:02.985893 8338 tablet_service.cc:1514] Processing DeleteTablet for tablet 58570133cd41478da075e8ccf0e5ebb7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 14:52:02 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:46842
I20260927 14:52:02.986014 8457 raft_consensus.cc:2229] T 58570133cd41478da075e8ccf0e5ebb7 P b9684a38ff254c1287b71a241d319172 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 14:52:02.986076 8455 tablet_replica.cc:330] T 58570133cd41478da075e8ccf0e5ebb7 P fd36717bc96647548d91ea87dd09dae4: stopping tablet replica
I20260927 14:52:02.986130 8457 raft_consensus.cc:2258] T 58570133cd41478da075e8ccf0e5ebb7 P b9684a38ff254c1287b71a241d319172 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 14:52:02.986187 8455 raft_consensus.cc:2229] T 58570133cd41478da075e8ccf0e5ebb7 P fd36717bc96647548d91ea87dd09dae4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 14:52:02.986291 8455 raft_consensus.cc:2258] T 58570133cd41478da075e8ccf0e5ebb7 P fd36717bc96647548d91ea87dd09dae4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 14:52:02.986299 8456 tablet_replica.cc:330] T 58570133cd41478da075e8ccf0e5ebb7 P 7a267c5c521e4912aad1c6124b1a92b6: stopping tablet replica
I20260927 14:52:02.986415 8456 raft_consensus.cc:2229] T 58570133cd41478da075e8ccf0e5ebb7 P 7a267c5c521e4912aad1c6124b1a92b6 [term 1 LEADER]: Raft consensus shutting down.
I20260927 14:52:02.986589 8456 raft_consensus.cc:2258] T 58570133cd41478da075e8ccf0e5ebb7 P 7a267c5c521e4912aad1c6124b1a92b6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 14:52:02.987269 8455 ts_tablet_manager.cc:1905] T 58570133cd41478da075e8ccf0e5ebb7 P fd36717bc96647548d91ea87dd09dae4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 14:52:02.987453 8457 ts_tablet_manager.cc:1905] T 58570133cd41478da075e8ccf0e5ebb7 P b9684a38ff254c1287b71a241d319172: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 14:52:02.989552 8456 ts_tablet_manager.cc:1905] T 58570133cd41478da075e8ccf0e5ebb7 P 7a267c5c521e4912aad1c6124b1a92b6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 14:52:02.990332 8455 ts_tablet_manager.cc:1918] T 58570133cd41478da075e8ccf0e5ebb7 P fd36717bc96647548d91ea87dd09dae4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 14:52:02.990437 8455 log.cc:1196] T 58570133cd41478da075e8ccf0e5ebb7 P fd36717bc96647548d91ea87dd09dae4: Deleting WAL directory at /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790520684602251-392-0/minicluster-data/ts-2/wal/wals/58570133cd41478da075e8ccf0e5ebb7
I20260927 14:52:02.990669 8455 ts_tablet_manager.cc:1939] T 58570133cd41478da075e8ccf0e5ebb7 P fd36717bc96647548d91ea87dd09dae4: Deleting consensus metadata
I20260927 14:52:02.991590 7905 catalog_manager.cc:4841] TS fd36717bc96647548d91ea87dd09dae4 (127.0.98.3:43169): tablet 58570133cd41478da075e8ccf0e5ebb7 (table test-table [id=30982364c9a3461f9102e519050999c1]) successfully deleted
I20260927 14:52:02.991981 8456 ts_tablet_manager.cc:1918] T 58570133cd41478da075e8ccf0e5ebb7 P 7a267c5c521e4912aad1c6124b1a92b6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 14:52:02.992069 8456 log.cc:1196] T 58570133cd41478da075e8ccf0e5ebb7 P 7a267c5c521e4912aad1c6124b1a92b6: Deleting WAL directory at /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790520684602251-392-0/minicluster-data/ts-0/wal/wals/58570133cd41478da075e8ccf0e5ebb7
I20260927 14:52:02.992317 8456 ts_tablet_manager.cc:1939] T 58570133cd41478da075e8ccf0e5ebb7 P 7a267c5c521e4912aad1c6124b1a92b6: Deleting consensus metadata
I20260927 14:52:02.992990 7905 catalog_manager.cc:4841] TS 7a267c5c521e4912aad1c6124b1a92b6 (127.0.98.1:46477): tablet 58570133cd41478da075e8ccf0e5ebb7 (table test-table [id=30982364c9a3461f9102e519050999c1]) successfully deleted
I20260927 14:52:02.994161 8457 ts_tablet_manager.cc:1918] T 58570133cd41478da075e8ccf0e5ebb7 P b9684a38ff254c1287b71a241d319172: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 14:52:02.994269 8457 log.cc:1196] T 58570133cd41478da075e8ccf0e5ebb7 P b9684a38ff254c1287b71a241d319172: Deleting WAL directory at /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790520684602251-392-0/minicluster-data/ts-1/wal/wals/58570133cd41478da075e8ccf0e5ebb7
I20260927 14:52:02.994515 8457 ts_tablet_manager.cc:1939] T 58570133cd41478da075e8ccf0e5ebb7 P b9684a38ff254c1287b71a241d319172: Deleting consensus metadata
I20260927 14:52:02.995185 7908 catalog_manager.cc:4841] TS b9684a38ff254c1287b71a241d319172 (127.0.98.2:34999): tablet 58570133cd41478da075e8ccf0e5ebb7 (table test-table [id=30982364c9a3461f9102e519050999c1]) successfully deleted
Sep 27 14:52:02 dist-test-slave-xk1g krb5kdc[7876](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790520722, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 27 14:52:03 dist-test-slave-xk1g krb5kdc[7876](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 14:52:03 dist-test-slave-xk1g krb5kdc[7876](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
W20260927 14:52:03.014088 8468 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 14:52:03.007226 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39088 (local address 127.0.98.62:42843)
0927 14:52:03.007370 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0927 14:52:03.007375 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 14:52:03.007538 (+ 163us) server_negotiation.cc:407] Connection header received
0927 14:52:03.007704 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:52:03.007707 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:52:03.007760 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:52:03.007851 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:52:03.008629 (+ 778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:52:03.009249 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:52:03.010245 (+ 996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:52:03.010489 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:52:03.010598 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:52:03.013965 (+ 3367us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39088: BlockingRecv error: recv got EOF from 127.0.0.1:39088 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
Sep 27 14:52:03 dist-test-slave-xk1g krb5kdc[7876](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790520722, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260927 14:52:03.027726 8468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:52:03.015635 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39104 (local address 127.0.98.62:42843)
0927 14:52:03.015742 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0927 14:52:03.015745 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 14:52:03.019338 (+ 3593us) server_negotiation.cc:407] Connection header received
0927 14:52:03.019502 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:52:03.019507 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:52:03.019560 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:52:03.019649 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:52:03.020238 (+ 589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:52:03.020755 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:52:03.021620 (+ 865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:52:03.021846 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:52:03.021973 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:52:03.024377 (+ 2404us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:52:03.024381 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:52:03.024383 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 14:52:03.024415 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:52:03.026028 (+ 1613us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:52:03.026461 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:52:03.026464 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:52:03.026466 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:52:03.026549 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:52:03.026829 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:52:03.026832 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:52:03.026834 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:52:03.027021 (+ 187us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:52:03.027162 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0927 14:52:03.027398 (+ 236us) server_negotiation.cc:299] Negotiation successful
0927 14:52:03.027533 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260927 14:52:03.030588 7920 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39104:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 14:52:03.031055 7920 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 14:52:03.035930 8058 tablet_service.cc:1467] Processing CreateTablet for tablet 75e0900ad2314278a99b1fb7b1016d3c (DEFAULT_TABLE table=test-table [id=c8297b0a30084c419ee9c11562f50ec4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 14:52:03.036219 8058 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 75e0900ad2314278a99b1fb7b1016d3c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 14:52:03.036346 8338 tablet_service.cc:1467] Processing CreateTablet for tablet 75e0900ad2314278a99b1fb7b1016d3c (DEFAULT_TABLE table=test-table [id=c8297b0a30084c419ee9c11562f50ec4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 14:52:03.036629 8338 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 75e0900ad2314278a99b1fb7b1016d3c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 14:52:03.038683 8434 tablet_bootstrap.cc:492] T 75e0900ad2314278a99b1fb7b1016d3c P fd36717bc96647548d91ea87dd09dae4: Bootstrap starting.
I20260927 14:52:03.038717 8198 tablet_service.cc:1467] Processing CreateTablet for tablet 75e0900ad2314278a99b1fb7b1016d3c (DEFAULT_TABLE table=test-table [id=c8297b0a30084c419ee9c11562f50ec4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 14:52:03.038986 8198 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 75e0900ad2314278a99b1fb7b1016d3c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 14:52:03.039461 8434 tablet_bootstrap.cc:654] T 75e0900ad2314278a99b1fb7b1016d3c P fd36717bc96647548d91ea87dd09dae4: Neither blocks nor log segments found. Creating new log.
I20260927 14:52:03.040725 8434 tablet_bootstrap.cc:492] T 75e0900ad2314278a99b1fb7b1016d3c P fd36717bc96647548d91ea87dd09dae4: No bootstrap required, opened a new log
I20260927 14:52:03.040825 8434 ts_tablet_manager.cc:1397] T 75e0900ad2314278a99b1fb7b1016d3c P fd36717bc96647548d91ea87dd09dae4: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 14:52:03.041361 8432 tablet_bootstrap.cc:492] T 75e0900ad2314278a99b1fb7b1016d3c P 7a267c5c521e4912aad1c6124b1a92b6: Bootstrap starting.
I20260927 14:52:03.041314 8434 raft_consensus.cc:348] T 75e0900ad2314278a99b1fb7b1016d3c P fd36717bc96647548d91ea87dd09dae4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a267c5c521e4912aad1c6124b1a92b6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46477 } } peers { permanent_uuid: "b9684a38ff254c1287b71a241d319172" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34999 } } peers { permanent_uuid: "fd36717bc96647548d91ea87dd09dae4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43169 } }
I20260927 14:52:03.041428 8434 raft_consensus.cc:374] T 75e0900ad2314278a99b1fb7b1016d3c P fd36717bc96647548d91ea87dd09dae4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 14:52:03.041468 8434 raft_consensus.cc:729] T 75e0900ad2314278a99b1fb7b1016d3c P fd36717bc96647548d91ea87dd09dae4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd36717bc96647548d91ea87dd09dae4, State: Initialized, Role: FOLLOWER
I20260927 14:52:03.041586 8434 consensus_queue.cc:260] T 75e0900ad2314278a99b1fb7b1016d3c P fd36717bc96647548d91ea87dd09dae4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a267c5c521e4912aad1c6124b1a92b6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46477 } } peers { permanent_uuid: "b9684a38ff254c1287b71a241d319172" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34999 } } peers { permanent_uuid: "fd36717bc96647548d91ea87dd09dae4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43169 } }
I20260927 14:52:03.041882 8434 ts_tablet_manager.cc:1428] T 75e0900ad2314278a99b1fb7b1016d3c P fd36717bc96647548d91ea87dd09dae4: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260927 14:52:03.042389 8432 tablet_bootstrap.cc:654] T 75e0900ad2314278a99b1fb7b1016d3c P 7a267c5c521e4912aad1c6124b1a92b6: Neither blocks nor log segments found. Creating new log.
I20260927 14:52:03.042951 8433 tablet_bootstrap.cc:492] T 75e0900ad2314278a99b1fb7b1016d3c P b9684a38ff254c1287b71a241d319172: Bootstrap starting.
I20260927 14:52:03.043756 8433 tablet_bootstrap.cc:654] T 75e0900ad2314278a99b1fb7b1016d3c P b9684a38ff254c1287b71a241d319172: Neither blocks nor log segments found. Creating new log.
I20260927 14:52:03.045094 8433 tablet_bootstrap.cc:492] T 75e0900ad2314278a99b1fb7b1016d3c P b9684a38ff254c1287b71a241d319172: No bootstrap required, opened a new log
I20260927 14:52:03.045184 8433 ts_tablet_manager.cc:1397] T 75e0900ad2314278a99b1fb7b1016d3c P b9684a38ff254c1287b71a241d319172: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260927 14:52:03.045840 8433 raft_consensus.cc:348] T 75e0900ad2314278a99b1fb7b1016d3c P b9684a38ff254c1287b71a241d319172 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a267c5c521e4912aad1c6124b1a92b6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46477 } } peers { permanent_uuid: "b9684a38ff254c1287b71a241d319172" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34999 } } peers { permanent_uuid: "fd36717bc96647548d91ea87dd09dae4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43169 } }
I20260927 14:52:03.045959 8433 raft_consensus.cc:374] T 75e0900ad2314278a99b1fb7b1016d3c P b9684a38ff254c1287b71a241d319172 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 14:52:03.045997 8433 raft_consensus.cc:729] T 75e0900ad2314278a99b1fb7b1016d3c P b9684a38ff254c1287b71a241d319172 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b9684a38ff254c1287b71a241d319172, State: Initialized, Role: FOLLOWER
I20260927 14:52:03.046128 8433 consensus_queue.cc:260] T 75e0900ad2314278a99b1fb7b1016d3c P b9684a38ff254c1287b71a241d319172 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a267c5c521e4912aad1c6124b1a92b6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46477 } } peers { permanent_uuid: "b9684a38ff254c1287b71a241d319172" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34999 } } peers { permanent_uuid: "fd36717bc96647548d91ea87dd09dae4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43169 } }
I20260927 14:52:03.046392 8433 ts_tablet_manager.cc:1428] T 75e0900ad2314278a99b1fb7b1016d3c P b9684a38ff254c1287b71a241d319172: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260927 14:52:03.047613 8432 tablet_bootstrap.cc:492] T 75e0900ad2314278a99b1fb7b1016d3c P 7a267c5c521e4912aad1c6124b1a92b6: No bootstrap required, opened a new log
I20260927 14:52:03.047709 8432 ts_tablet_manager.cc:1397] T 75e0900ad2314278a99b1fb7b1016d3c P 7a267c5c521e4912aad1c6124b1a92b6: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20260927 14:52:03.048130 8432 raft_consensus.cc:348] T 75e0900ad2314278a99b1fb7b1016d3c P 7a267c5c521e4912aad1c6124b1a92b6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a267c5c521e4912aad1c6124b1a92b6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46477 } } peers { permanent_uuid: "b9684a38ff254c1287b71a241d319172" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34999 } } peers { permanent_uuid: "fd36717bc96647548d91ea87dd09dae4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43169 } }
I20260927 14:52:03.048252 8432 raft_consensus.cc:374] T 75e0900ad2314278a99b1fb7b1016d3c P 7a267c5c521e4912aad1c6124b1a92b6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 14:52:03.048298 8432 raft_consensus.cc:729] T 75e0900ad2314278a99b1fb7b1016d3c P 7a267c5c521e4912aad1c6124b1a92b6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a267c5c521e4912aad1c6124b1a92b6, State: Initialized, Role: FOLLOWER
I20260927 14:52:03.048429 8432 consensus_queue.cc:260] T 75e0900ad2314278a99b1fb7b1016d3c P 7a267c5c521e4912aad1c6124b1a92b6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a267c5c521e4912aad1c6124b1a92b6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46477 } } peers { permanent_uuid: "b9684a38ff254c1287b71a241d319172" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34999 } } peers { permanent_uuid: "fd36717bc96647548d91ea87dd09dae4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43169 } }
I20260927 14:52:03.048707 8432 ts_tablet_manager.cc:1428] T 75e0900ad2314278a99b1fb7b1016d3c P 7a267c5c521e4912aad1c6124b1a92b6: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260927 14:52:03.088546 8478 raft_consensus.cc:482] T 75e0900ad2314278a99b1fb7b1016d3c P 7a267c5c521e4912aad1c6124b1a92b6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 14:52:03.088688 8478 raft_consensus.cc:504] T 75e0900ad2314278a99b1fb7b1016d3c P 7a267c5c521e4912aad1c6124b1a92b6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a267c5c521e4912aad1c6124b1a92b6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46477 } } peers { permanent_uuid: "b9684a38ff254c1287b71a241d319172" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34999 } } peers { permanent_uuid: "fd36717bc96647548d91ea87dd09dae4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43169 } }
I20260927 14:52:03.089165 8478 leader_election.cc:288] T 75e0900ad2314278a99b1fb7b1016d3c P 7a267c5c521e4912aad1c6124b1a92b6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b9684a38ff254c1287b71a241d319172 (127.0.98.2:34999), fd36717bc96647548d91ea87dd09dae4 (127.0.98.3:43169)
I20260927 14:52:03.089545 8218 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "75e0900ad2314278a99b1fb7b1016d3c" candidate_uuid: "7a267c5c521e4912aad1c6124b1a92b6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9684a38ff254c1287b71a241d319172" is_pre_election: true
I20260927 14:52:03.089579 8357 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "75e0900ad2314278a99b1fb7b1016d3c" candidate_uuid: "7a267c5c521e4912aad1c6124b1a92b6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd36717bc96647548d91ea87dd09dae4" is_pre_election: true
I20260927 14:52:03.089675 8218 raft_consensus.cc:2454] T 75e0900ad2314278a99b1fb7b1016d3c P b9684a38ff254c1287b71a241d319172 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a267c5c521e4912aad1c6124b1a92b6 in term 0.
I20260927 14:52:03.089741 8357 raft_consensus.cc:2454] T 75e0900ad2314278a99b1fb7b1016d3c P fd36717bc96647548d91ea87dd09dae4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a267c5c521e4912aad1c6124b1a92b6 in term 0.
I20260927 14:52:03.090045 8012 leader_election.cc:302] T 75e0900ad2314278a99b1fb7b1016d3c P 7a267c5c521e4912aad1c6124b1a92b6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a267c5c521e4912aad1c6124b1a92b6, b9684a38ff254c1287b71a241d319172; no voters:
I20260927 14:52:03.090241 8478 raft_consensus.cc:2781] T 75e0900ad2314278a99b1fb7b1016d3c P 7a267c5c521e4912aad1c6124b1a92b6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 14:52:03.090312 8478 raft_consensus.cc:482] T 75e0900ad2314278a99b1fb7b1016d3c P 7a267c5c521e4912aad1c6124b1a92b6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 14:52:03.090364 8478 raft_consensus.cc:3037] T 75e0900ad2314278a99b1fb7b1016d3c P 7a267c5c521e4912aad1c6124b1a92b6 [term 0 FOLLOWER]: Advancing to term 1
I20260927 14:52:03.091349 8478 raft_consensus.cc:504] T 75e0900ad2314278a99b1fb7b1016d3c P 7a267c5c521e4912aad1c6124b1a92b6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a267c5c521e4912aad1c6124b1a92b6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46477 } } peers { permanent_uuid: "b9684a38ff254c1287b71a241d319172" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34999 } } peers { permanent_uuid: "fd36717bc96647548d91ea87dd09dae4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43169 } }
I20260927 14:52:03.091682 8478 leader_election.cc:288] T 75e0900ad2314278a99b1fb7b1016d3c P 7a267c5c521e4912aad1c6124b1a92b6 [CANDIDATE]: Term 1 election: Requested vote from peers b9684a38ff254c1287b71a241d319172 (127.0.98.2:34999), fd36717bc96647548d91ea87dd09dae4 (127.0.98.3:43169)
I20260927 14:52:03.092041 8357 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "75e0900ad2314278a99b1fb7b1016d3c" candidate_uuid: "7a267c5c521e4912aad1c6124b1a92b6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd36717bc96647548d91ea87dd09dae4"
I20260927 14:52:03.092063 8218 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "75e0900ad2314278a99b1fb7b1016d3c" candidate_uuid: "7a267c5c521e4912aad1c6124b1a92b6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9684a38ff254c1287b71a241d319172"
I20260927 14:52:03.092175 8357 raft_consensus.cc:3037] T 75e0900ad2314278a99b1fb7b1016d3c P fd36717bc96647548d91ea87dd09dae4 [term 0 FOLLOWER]: Advancing to term 1
I20260927 14:52:03.092188 8218 raft_consensus.cc:3037] T 75e0900ad2314278a99b1fb7b1016d3c P b9684a38ff254c1287b71a241d319172 [term 0 FOLLOWER]: Advancing to term 1
I20260927 14:52:03.093109 8218 raft_consensus.cc:2454] T 75e0900ad2314278a99b1fb7b1016d3c P b9684a38ff254c1287b71a241d319172 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a267c5c521e4912aad1c6124b1a92b6 in term 1.
I20260927 14:52:03.093397 8012 leader_election.cc:302] T 75e0900ad2314278a99b1fb7b1016d3c P 7a267c5c521e4912aad1c6124b1a92b6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a267c5c521e4912aad1c6124b1a92b6, b9684a38ff254c1287b71a241d319172; no voters:
I20260927 14:52:03.093566 8478 raft_consensus.cc:2781] T 75e0900ad2314278a99b1fb7b1016d3c P 7a267c5c521e4912aad1c6124b1a92b6 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 14:52:03.093654 8478 raft_consensus.cc:686] T 75e0900ad2314278a99b1fb7b1016d3c P 7a267c5c521e4912aad1c6124b1a92b6 [term 1 LEADER]: Becoming Leader. State: Replica: 7a267c5c521e4912aad1c6124b1a92b6, State: Running, Role: LEADER
I20260927 14:52:03.093690 8357 raft_consensus.cc:2454] T 75e0900ad2314278a99b1fb7b1016d3c P fd36717bc96647548d91ea87dd09dae4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a267c5c521e4912aad1c6124b1a92b6 in term 1.
I20260927 14:52:03.093792 8478 consensus_queue.cc:237] T 75e0900ad2314278a99b1fb7b1016d3c P 7a267c5c521e4912aad1c6124b1a92b6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a267c5c521e4912aad1c6124b1a92b6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46477 } } peers { permanent_uuid: "b9684a38ff254c1287b71a241d319172" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34999 } } peers { permanent_uuid: "fd36717bc96647548d91ea87dd09dae4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43169 } }
I20260927 14:52:03.095329 7921 catalog_manager.cc:5495] T 75e0900ad2314278a99b1fb7b1016d3c P 7a267c5c521e4912aad1c6124b1a92b6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7a267c5c521e4912aad1c6124b1a92b6 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "7a267c5c521e4912aad1c6124b1a92b6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a267c5c521e4912aad1c6124b1a92b6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46477 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b9684a38ff254c1287b71a241d319172" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34999 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fd36717bc96647548d91ea87dd09dae4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43169 } health_report { overall_health: UNKNOWN } } }
I20260927 14:52:03.103832 8430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:52:03.100943 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56624 (local address 127.0.98.1:46477)
0927 14:52:03.101127 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0927 14:52:03.101133 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 14:52:03.101185 (+ 52us) server_negotiation.cc:407] Connection header received
0927 14:52:03.101269 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:52:03.101271 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:52:03.101313 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:52:03.101373 (+ 60us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 14:52:03.101580 (+ 207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:52:03.102153 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:52:03.102974 (+ 821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:52:03.103177 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:52:03.103252 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:52:03.103452 (+ 200us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 14:52:03.103524 (+ 72us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 14:52:03.103574 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0927 14:52:03.103636 (+ 62us) server_negotiation.cc:299] Negotiation successful
0927 14:52:03.103676 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92}
I20260927 14:52:03.105593 8218 raft_consensus.cc:1261] T 75e0900ad2314278a99b1fb7b1016d3c P b9684a38ff254c1287b71a241d319172 [term 1 FOLLOWER]: Refusing update from remote peer 7a267c5c521e4912aad1c6124b1a92b6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 14:52:03.105593 8357 raft_consensus.cc:1261] T 75e0900ad2314278a99b1fb7b1016d3c P fd36717bc96647548d91ea87dd09dae4 [term 1 FOLLOWER]: Refusing update from remote peer 7a267c5c521e4912aad1c6124b1a92b6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 14:52:03.105995 8478 consensus_queue.cc:1035] T 75e0900ad2314278a99b1fb7b1016d3c P 7a267c5c521e4912aad1c6124b1a92b6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9684a38ff254c1287b71a241d319172" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34999 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 14:52:03.106122 8485 consensus_queue.cc:1035] T 75e0900ad2314278a99b1fb7b1016d3c P 7a267c5c521e4912aad1c6124b1a92b6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fd36717bc96647548d91ea87dd09dae4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43169 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 14:52:03.114833 7921 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39104:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 14:52:03.114980 7921 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39104:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 14:52:03.116731 7921 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d4bb90ee61a346e783b1407c8f06dc21: Sending DeleteTablet for 3 replicas of tablet 75e0900ad2314278a99b1fb7b1016d3c
I20260927 14:52:03.117255 8058 tablet_service.cc:1514] Processing DeleteTablet for tablet 75e0900ad2314278a99b1fb7b1016d3c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 14:52:03 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:56588
I20260927 14:52:03.117345 8338 tablet_service.cc:1514] Processing DeleteTablet for tablet 75e0900ad2314278a99b1fb7b1016d3c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 14:52:03 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:46842
I20260927 14:52:03.117452 8456 tablet_replica.cc:330] T 75e0900ad2314278a99b1fb7b1016d3c P 7a267c5c521e4912aad1c6124b1a92b6: stopping tablet replica
I20260927 14:52:03.117518 8455 tablet_replica.cc:330] T 75e0900ad2314278a99b1fb7b1016d3c P fd36717bc96647548d91ea87dd09dae4: stopping tablet replica
I20260927 14:52:03.117606 8456 raft_consensus.cc:2229] T 75e0900ad2314278a99b1fb7b1016d3c P 7a267c5c521e4912aad1c6124b1a92b6 [term 1 LEADER]: Raft consensus shutting down.
I20260927 14:52:03.117647 8455 raft_consensus.cc:2229] T 75e0900ad2314278a99b1fb7b1016d3c P fd36717bc96647548d91ea87dd09dae4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 14:52:03.117775 8455 raft_consensus.cc:2258] T 75e0900ad2314278a99b1fb7b1016d3c P fd36717bc96647548d91ea87dd09dae4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 14:52:03.117813 8456 raft_consensus.cc:2258] T 75e0900ad2314278a99b1fb7b1016d3c P 7a267c5c521e4912aad1c6124b1a92b6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 14:52:03.118086 8198 tablet_service.cc:1514] Processing DeleteTablet for tablet 75e0900ad2314278a99b1fb7b1016d3c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 14:52:03 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:52474
I20260927 14:52:03.118099 7921 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39104:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260927 14:52:03.118251 8457 tablet_replica.cc:330] T 75e0900ad2314278a99b1fb7b1016d3c P b9684a38ff254c1287b71a241d319172: stopping tablet replica
I20260927 14:52:03.118338 8457 raft_consensus.cc:2229] T 75e0900ad2314278a99b1fb7b1016d3c P b9684a38ff254c1287b71a241d319172 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 14:52:03.118449 8457 raft_consensus.cc:2258] T 75e0900ad2314278a99b1fb7b1016d3c P b9684a38ff254c1287b71a241d319172 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260927 14:52:03.118472 7921 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 14:52:03.118502 8456 ts_tablet_manager.cc:1905] T 75e0900ad2314278a99b1fb7b1016d3c P 7a267c5c521e4912aad1c6124b1a92b6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 14:52:03.119032 8455 ts_tablet_manager.cc:1905] T 75e0900ad2314278a99b1fb7b1016d3c P fd36717bc96647548d91ea87dd09dae4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 14:52:03.120400 8456 ts_tablet_manager.cc:1918] T 75e0900ad2314278a99b1fb7b1016d3c P 7a267c5c521e4912aad1c6124b1a92b6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 14:52:03.120488 8456 log.cc:1196] T 75e0900ad2314278a99b1fb7b1016d3c P 7a267c5c521e4912aad1c6124b1a92b6: Deleting WAL directory at /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790520684602251-392-0/minicluster-data/ts-0/wal/wals/75e0900ad2314278a99b1fb7b1016d3c
I20260927 14:52:03.120739 8456 ts_tablet_manager.cc:1939] T 75e0900ad2314278a99b1fb7b1016d3c P 7a267c5c521e4912aad1c6124b1a92b6: Deleting consensus metadata
I20260927 14:52:03.120965 8455 ts_tablet_manager.cc:1918] T 75e0900ad2314278a99b1fb7b1016d3c P fd36717bc96647548d91ea87dd09dae4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 14:52:03.121028 8455 log.cc:1196] T 75e0900ad2314278a99b1fb7b1016d3c P fd36717bc96647548d91ea87dd09dae4: Deleting WAL directory at /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790520684602251-392-0/minicluster-data/ts-2/wal/wals/75e0900ad2314278a99b1fb7b1016d3c
I20260927 14:52:03.121333 8455 ts_tablet_manager.cc:1939] T 75e0900ad2314278a99b1fb7b1016d3c P fd36717bc96647548d91ea87dd09dae4: Deleting consensus metadata
I20260927 14:52:03.121397 7905 catalog_manager.cc:4841] TS 7a267c5c521e4912aad1c6124b1a92b6 (127.0.98.1:46477): tablet 75e0900ad2314278a99b1fb7b1016d3c (table test-table [id=c8297b0a30084c419ee9c11562f50ec4]) successfully deleted
I20260927 14:52:03.121977 7905 catalog_manager.cc:4841] TS fd36717bc96647548d91ea87dd09dae4 (127.0.98.3:43169): tablet 75e0900ad2314278a99b1fb7b1016d3c (table test-table [id=c8297b0a30084c419ee9c11562f50ec4]) successfully deleted
I20260927 14:52:03.122179 8457 ts_tablet_manager.cc:1905] T 75e0900ad2314278a99b1fb7b1016d3c P b9684a38ff254c1287b71a241d319172: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 14:52:03.124027 8457 ts_tablet_manager.cc:1918] T 75e0900ad2314278a99b1fb7b1016d3c P b9684a38ff254c1287b71a241d319172: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 14:52:03.124089 8457 log.cc:1196] T 75e0900ad2314278a99b1fb7b1016d3c P b9684a38ff254c1287b71a241d319172: Deleting WAL directory at /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790520684602251-392-0/minicluster-data/ts-1/wal/wals/75e0900ad2314278a99b1fb7b1016d3c
I20260927 14:52:03.124322 8457 ts_tablet_manager.cc:1939] T 75e0900ad2314278a99b1fb7b1016d3c P b9684a38ff254c1287b71a241d319172: Deleting consensus metadata
I20260927 14:52:03.125043 7908 catalog_manager.cc:4841] TS b9684a38ff254c1287b71a241d319172 (127.0.98.2:34999): tablet 75e0900ad2314278a99b1fb7b1016d3c (table test-table [id=c8297b0a30084c419ee9c11562f50ec4]) successfully deleted
I20260927 14:52:03.128114 8058 tablet_service.cc:1467] Processing CreateTablet for tablet 5bc2c777a3184f8caeeb5be6077c5dd0 (DEFAULT_TABLE table=test-table [id=a8169a72411f4577a4a269078a019c36]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 14:52:03.128185 8198 tablet_service.cc:1467] Processing CreateTablet for tablet 5bc2c777a3184f8caeeb5be6077c5dd0 (DEFAULT_TABLE table=test-table [id=a8169a72411f4577a4a269078a019c36]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 14:52:03.128304 8338 tablet_service.cc:1467] Processing CreateTablet for tablet 5bc2c777a3184f8caeeb5be6077c5dd0 (DEFAULT_TABLE table=test-table [id=a8169a72411f4577a4a269078a019c36]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 14:52:03.128374 8058 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5bc2c777a3184f8caeeb5be6077c5dd0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 14:52:03.128453 8198 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5bc2c777a3184f8caeeb5be6077c5dd0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 14:52:03.128537 8338 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5bc2c777a3184f8caeeb5be6077c5dd0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 14:52:03.130606 8433 tablet_bootstrap.cc:492] T 5bc2c777a3184f8caeeb5be6077c5dd0 P b9684a38ff254c1287b71a241d319172: Bootstrap starting.
I20260927 14:52:03.130764 8434 tablet_bootstrap.cc:492] T 5bc2c777a3184f8caeeb5be6077c5dd0 P fd36717bc96647548d91ea87dd09dae4: Bootstrap starting.
I20260927 14:52:03.130967 8432 tablet_bootstrap.cc:492] T 5bc2c777a3184f8caeeb5be6077c5dd0 P 7a267c5c521e4912aad1c6124b1a92b6: Bootstrap starting.
I20260927 14:52:03.131623 8432 tablet_bootstrap.cc:654] T 5bc2c777a3184f8caeeb5be6077c5dd0 P 7a267c5c521e4912aad1c6124b1a92b6: Neither blocks nor log segments found. Creating new log.
I20260927 14:52:03.131628 8433 tablet_bootstrap.cc:654] T 5bc2c777a3184f8caeeb5be6077c5dd0 P b9684a38ff254c1287b71a241d319172: Neither blocks nor log segments found. Creating new log.
I20260927 14:52:03.131628 8434 tablet_bootstrap.cc:654] T 5bc2c777a3184f8caeeb5be6077c5dd0 P fd36717bc96647548d91ea87dd09dae4: Neither blocks nor log segments found. Creating new log.
I20260927 14:52:03.132618 8432 tablet_bootstrap.cc:492] T 5bc2c777a3184f8caeeb5be6077c5dd0 P 7a267c5c521e4912aad1c6124b1a92b6: No bootstrap required, opened a new log
I20260927 14:52:03.132745 8432 ts_tablet_manager.cc:1397] T 5bc2c777a3184f8caeeb5be6077c5dd0 P 7a267c5c521e4912aad1c6124b1a92b6: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 14:52:03.132936 8434 tablet_bootstrap.cc:492] T 5bc2c777a3184f8caeeb5be6077c5dd0 P fd36717bc96647548d91ea87dd09dae4: No bootstrap required, opened a new log
I20260927 14:52:03.133029 8434 ts_tablet_manager.cc:1397] T 5bc2c777a3184f8caeeb5be6077c5dd0 P fd36717bc96647548d91ea87dd09dae4: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260927 14:52:03.133210 8432 raft_consensus.cc:348] T 5bc2c777a3184f8caeeb5be6077c5dd0 P 7a267c5c521e4912aad1c6124b1a92b6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a267c5c521e4912aad1c6124b1a92b6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46477 } } peers { permanent_uuid: "b9684a38ff254c1287b71a241d319172" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34999 } } peers { permanent_uuid: "fd36717bc96647548d91ea87dd09dae4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43169 } }
I20260927 14:52:03.133324 8432 raft_consensus.cc:374] T 5bc2c777a3184f8caeeb5be6077c5dd0 P 7a267c5c521e4912aad1c6124b1a92b6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 14:52:03.133368 8432 raft_consensus.cc:729] T 5bc2c777a3184f8caeeb5be6077c5dd0 P 7a267c5c521e4912aad1c6124b1a92b6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a267c5c521e4912aad1c6124b1a92b6, State: Initialized, Role: FOLLOWER
I20260927 14:52:03.133459 8434 raft_consensus.cc:348] T 5bc2c777a3184f8caeeb5be6077c5dd0 P fd36717bc96647548d91ea87dd09dae4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a267c5c521e4912aad1c6124b1a92b6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46477 } } peers { permanent_uuid: "b9684a38ff254c1287b71a241d319172" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34999 } } peers { permanent_uuid: "fd36717bc96647548d91ea87dd09dae4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43169 } }
I20260927 14:52:03.133491 8432 consensus_queue.cc:260] T 5bc2c777a3184f8caeeb5be6077c5dd0 P 7a267c5c521e4912aad1c6124b1a92b6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a267c5c521e4912aad1c6124b1a92b6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46477 } } peers { permanent_uuid: "b9684a38ff254c1287b71a241d319172" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34999 } } peers { permanent_uuid: "fd36717bc96647548d91ea87dd09dae4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43169 } }
I20260927 14:52:03.133562 8434 raft_consensus.cc:374] T 5bc2c777a3184f8caeeb5be6077c5dd0 P fd36717bc96647548d91ea87dd09dae4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 14:52:03.133584 8434 raft_consensus.cc:729] T 5bc2c777a3184f8caeeb5be6077c5dd0 P fd36717bc96647548d91ea87dd09dae4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd36717bc96647548d91ea87dd09dae4, State: Initialized, Role: FOLLOWER
I20260927 14:52:03.133725 8434 consensus_queue.cc:260] T 5bc2c777a3184f8caeeb5be6077c5dd0 P fd36717bc96647548d91ea87dd09dae4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a267c5c521e4912aad1c6124b1a92b6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46477 } } peers { permanent_uuid: "b9684a38ff254c1287b71a241d319172" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34999 } } peers { permanent_uuid: "fd36717bc96647548d91ea87dd09dae4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43169 } }
I20260927 14:52:03.133832 8432 ts_tablet_manager.cc:1428] T 5bc2c777a3184f8caeeb5be6077c5dd0 P 7a267c5c521e4912aad1c6124b1a92b6: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260927 14:52:03.133987 8434 ts_tablet_manager.cc:1428] T 5bc2c777a3184f8caeeb5be6077c5dd0 P fd36717bc96647548d91ea87dd09dae4: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260927 14:52:03.134950 8433 tablet_bootstrap.cc:492] T 5bc2c777a3184f8caeeb5be6077c5dd0 P b9684a38ff254c1287b71a241d319172: No bootstrap required, opened a new log
I20260927 14:52:03.135030 8433 ts_tablet_manager.cc:1397] T 5bc2c777a3184f8caeeb5be6077c5dd0 P b9684a38ff254c1287b71a241d319172: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260927 14:52:03.135885 8433 raft_consensus.cc:348] T 5bc2c777a3184f8caeeb5be6077c5dd0 P b9684a38ff254c1287b71a241d319172 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a267c5c521e4912aad1c6124b1a92b6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46477 } } peers { permanent_uuid: "b9684a38ff254c1287b71a241d319172" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34999 } } peers { permanent_uuid: "fd36717bc96647548d91ea87dd09dae4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43169 } }
I20260927 14:52:03.136016 8433 raft_consensus.cc:374] T 5bc2c777a3184f8caeeb5be6077c5dd0 P b9684a38ff254c1287b71a241d319172 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 14:52:03.136056 8433 raft_consensus.cc:729] T 5bc2c777a3184f8caeeb5be6077c5dd0 P b9684a38ff254c1287b71a241d319172 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b9684a38ff254c1287b71a241d319172, State: Initialized, Role: FOLLOWER
I20260927 14:52:03.136188 8433 consensus_queue.cc:260] T 5bc2c777a3184f8caeeb5be6077c5dd0 P b9684a38ff254c1287b71a241d319172 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a267c5c521e4912aad1c6124b1a92b6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46477 } } peers { permanent_uuid: "b9684a38ff254c1287b71a241d319172" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34999 } } peers { permanent_uuid: "fd36717bc96647548d91ea87dd09dae4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43169 } }
I20260927 14:52:03.136451 8433 ts_tablet_manager.cc:1428] T 5bc2c777a3184f8caeeb5be6077c5dd0 P b9684a38ff254c1287b71a241d319172: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260927 14:52:03.332342 8485 raft_consensus.cc:482] T 5bc2c777a3184f8caeeb5be6077c5dd0 P 7a267c5c521e4912aad1c6124b1a92b6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 14:52:03.332496 8485 raft_consensus.cc:504] T 5bc2c777a3184f8caeeb5be6077c5dd0 P 7a267c5c521e4912aad1c6124b1a92b6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a267c5c521e4912aad1c6124b1a92b6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46477 } } peers { permanent_uuid: "b9684a38ff254c1287b71a241d319172" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34999 } } peers { permanent_uuid: "fd36717bc96647548d91ea87dd09dae4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43169 } }
I20260927 14:52:03.333169 8485 leader_election.cc:288] T 5bc2c777a3184f8caeeb5be6077c5dd0 P 7a267c5c521e4912aad1c6124b1a92b6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b9684a38ff254c1287b71a241d319172 (127.0.98.2:34999), fd36717bc96647548d91ea87dd09dae4 (127.0.98.3:43169)
I20260927 14:52:03.333616 8218 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5bc2c777a3184f8caeeb5be6077c5dd0" candidate_uuid: "7a267c5c521e4912aad1c6124b1a92b6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9684a38ff254c1287b71a241d319172" is_pre_election: true
I20260927 14:52:03.333693 8357 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5bc2c777a3184f8caeeb5be6077c5dd0" candidate_uuid: "7a267c5c521e4912aad1c6124b1a92b6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd36717bc96647548d91ea87dd09dae4" is_pre_election: true
I20260927 14:52:03.333794 8218 raft_consensus.cc:2454] T 5bc2c777a3184f8caeeb5be6077c5dd0 P b9684a38ff254c1287b71a241d319172 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a267c5c521e4912aad1c6124b1a92b6 in term 0.
I20260927 14:52:03.333899 8357 raft_consensus.cc:2454] T 5bc2c777a3184f8caeeb5be6077c5dd0 P fd36717bc96647548d91ea87dd09dae4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a267c5c521e4912aad1c6124b1a92b6 in term 0.
I20260927 14:52:03.334158 8012 leader_election.cc:302] T 5bc2c777a3184f8caeeb5be6077c5dd0 P 7a267c5c521e4912aad1c6124b1a92b6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a267c5c521e4912aad1c6124b1a92b6, b9684a38ff254c1287b71a241d319172; no voters:
I20260927 14:52:03.334331 8485 raft_consensus.cc:2781] T 5bc2c777a3184f8caeeb5be6077c5dd0 P 7a267c5c521e4912aad1c6124b1a92b6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 14:52:03.334390 8485 raft_consensus.cc:482] T 5bc2c777a3184f8caeeb5be6077c5dd0 P 7a267c5c521e4912aad1c6124b1a92b6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 14:52:03.334444 8485 raft_consensus.cc:3037] T 5bc2c777a3184f8caeeb5be6077c5dd0 P 7a267c5c521e4912aad1c6124b1a92b6 [term 0 FOLLOWER]: Advancing to term 1
I20260927 14:52:03.335474 8485 raft_consensus.cc:504] T 5bc2c777a3184f8caeeb5be6077c5dd0 P 7a267c5c521e4912aad1c6124b1a92b6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a267c5c521e4912aad1c6124b1a92b6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46477 } } peers { permanent_uuid: "b9684a38ff254c1287b71a241d319172" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34999 } } peers { permanent_uuid: "fd36717bc96647548d91ea87dd09dae4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43169 } }
I20260927 14:52:03.335844 8485 leader_election.cc:288] T 5bc2c777a3184f8caeeb5be6077c5dd0 P 7a267c5c521e4912aad1c6124b1a92b6 [CANDIDATE]: Term 1 election: Requested vote from peers b9684a38ff254c1287b71a241d319172 (127.0.98.2:34999), fd36717bc96647548d91ea87dd09dae4 (127.0.98.3:43169)
I20260927 14:52:03.336150 8218 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5bc2c777a3184f8caeeb5be6077c5dd0" candidate_uuid: "7a267c5c521e4912aad1c6124b1a92b6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9684a38ff254c1287b71a241d319172"
I20260927 14:52:03.336215 8357 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5bc2c777a3184f8caeeb5be6077c5dd0" candidate_uuid: "7a267c5c521e4912aad1c6124b1a92b6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd36717bc96647548d91ea87dd09dae4"
I20260927 14:52:03.336268 8218 raft_consensus.cc:3037] T 5bc2c777a3184f8caeeb5be6077c5dd0 P b9684a38ff254c1287b71a241d319172 [term 0 FOLLOWER]: Advancing to term 1
I20260927 14:52:03.336350 8357 raft_consensus.cc:3037] T 5bc2c777a3184f8caeeb5be6077c5dd0 P fd36717bc96647548d91ea87dd09dae4 [term 0 FOLLOWER]: Advancing to term 1
I20260927 14:52:03.337239 8218 raft_consensus.cc:2454] T 5bc2c777a3184f8caeeb5be6077c5dd0 P b9684a38ff254c1287b71a241d319172 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a267c5c521e4912aad1c6124b1a92b6 in term 1.
I20260927 14:52:03.337397 8357 raft_consensus.cc:2454] T 5bc2c777a3184f8caeeb5be6077c5dd0 P fd36717bc96647548d91ea87dd09dae4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a267c5c521e4912aad1c6124b1a92b6 in term 1.
I20260927 14:52:03.337536 8012 leader_election.cc:302] T 5bc2c777a3184f8caeeb5be6077c5dd0 P 7a267c5c521e4912aad1c6124b1a92b6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a267c5c521e4912aad1c6124b1a92b6, b9684a38ff254c1287b71a241d319172; no voters:
I20260927 14:52:03.337707 8485 raft_consensus.cc:2781] T 5bc2c777a3184f8caeeb5be6077c5dd0 P 7a267c5c521e4912aad1c6124b1a92b6 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 14:52:03.337818 8485 raft_consensus.cc:686] T 5bc2c777a3184f8caeeb5be6077c5dd0 P 7a267c5c521e4912aad1c6124b1a92b6 [term 1 LEADER]: Becoming Leader. State: Replica: 7a267c5c521e4912aad1c6124b1a92b6, State: Running, Role: LEADER
I20260927 14:52:03.337992 8485 consensus_queue.cc:237] T 5bc2c777a3184f8caeeb5be6077c5dd0 P 7a267c5c521e4912aad1c6124b1a92b6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a267c5c521e4912aad1c6124b1a92b6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46477 } } peers { permanent_uuid: "b9684a38ff254c1287b71a241d319172" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34999 } } peers { permanent_uuid: "fd36717bc96647548d91ea87dd09dae4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43169 } }
I20260927 14:52:03.339709 7921 catalog_manager.cc:5495] T 5bc2c777a3184f8caeeb5be6077c5dd0 P 7a267c5c521e4912aad1c6124b1a92b6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7a267c5c521e4912aad1c6124b1a92b6 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "7a267c5c521e4912aad1c6124b1a92b6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a267c5c521e4912aad1c6124b1a92b6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46477 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b9684a38ff254c1287b71a241d319172" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34999 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fd36717bc96647548d91ea87dd09dae4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43169 } health_report { overall_health: UNKNOWN } } }
I20260927 14:52:03.378847 8218 raft_consensus.cc:1261] T 5bc2c777a3184f8caeeb5be6077c5dd0 P b9684a38ff254c1287b71a241d319172 [term 1 FOLLOWER]: Refusing update from remote peer 7a267c5c521e4912aad1c6124b1a92b6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 14:52:03.378896 8357 raft_consensus.cc:1261] T 5bc2c777a3184f8caeeb5be6077c5dd0 P fd36717bc96647548d91ea87dd09dae4 [term 1 FOLLOWER]: Refusing update from remote peer 7a267c5c521e4912aad1c6124b1a92b6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 14:52:03.379211 8485 consensus_queue.cc:1035] T 5bc2c777a3184f8caeeb5be6077c5dd0 P 7a267c5c521e4912aad1c6124b1a92b6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9684a38ff254c1287b71a241d319172" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34999 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 14:52:03.379377 8478 consensus_queue.cc:1035] T 5bc2c777a3184f8caeeb5be6077c5dd0 P 7a267c5c521e4912aad1c6124b1a92b6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fd36717bc96647548d91ea87dd09dae4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43169 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 14:52:03.405869 7921 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39104:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 14:52:03.406050 7921 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39104:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 14:52:03.407826 7921 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d4bb90ee61a346e783b1407c8f06dc21: Sending DeleteTablet for 3 replicas of tablet 5bc2c777a3184f8caeeb5be6077c5dd0
I20260927 14:52:03.408272 8057 tablet_service.cc:1514] Processing DeleteTablet for tablet 5bc2c777a3184f8caeeb5be6077c5dd0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 14:52:03 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:56588
I20260927 14:52:03.408361 8198 tablet_service.cc:1514] Processing DeleteTablet for tablet 5bc2c777a3184f8caeeb5be6077c5dd0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 14:52:03 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:52474
I20260927 14:52:03.408444 8456 tablet_replica.cc:330] T 5bc2c777a3184f8caeeb5be6077c5dd0 P 7a267c5c521e4912aad1c6124b1a92b6: stopping tablet replica
I20260927 14:52:03.408439 8338 tablet_service.cc:1514] Processing DeleteTablet for tablet 5bc2c777a3184f8caeeb5be6077c5dd0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 14:52:03 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:46842
I20260927 14:52:03.408556 8456 raft_consensus.cc:2229] T 5bc2c777a3184f8caeeb5be6077c5dd0 P 7a267c5c521e4912aad1c6124b1a92b6 [term 1 LEADER]: Raft consensus shutting down.
I20260927 14:52:03.408586 8457 tablet_replica.cc:330] T 5bc2c777a3184f8caeeb5be6077c5dd0 P b9684a38ff254c1287b71a241d319172: stopping tablet replica
I20260927 14:52:03.408679 8457 raft_consensus.cc:2229] T 5bc2c777a3184f8caeeb5be6077c5dd0 P b9684a38ff254c1287b71a241d319172 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 14:52:03.408751 8456 raft_consensus.cc:2258] T 5bc2c777a3184f8caeeb5be6077c5dd0 P 7a267c5c521e4912aad1c6124b1a92b6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 14:52:03.408772 8457 raft_consensus.cc:2258] T 5bc2c777a3184f8caeeb5be6077c5dd0 P b9684a38ff254c1287b71a241d319172 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 14:52:03.408918 8455 tablet_replica.cc:330] T 5bc2c777a3184f8caeeb5be6077c5dd0 P fd36717bc96647548d91ea87dd09dae4: stopping tablet replica
I20260927 14:52:03.409049 8455 raft_consensus.cc:2229] T 5bc2c777a3184f8caeeb5be6077c5dd0 P fd36717bc96647548d91ea87dd09dae4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 14:52:03.409200 8455 raft_consensus.cc:2258] T 5bc2c777a3184f8caeeb5be6077c5dd0 P fd36717bc96647548d91ea87dd09dae4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 14:52:03.409490 8456 ts_tablet_manager.cc:1905] T 5bc2c777a3184f8caeeb5be6077c5dd0 P 7a267c5c521e4912aad1c6124b1a92b6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 14:52:03.409879 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6936m5/build/debug/bin/kudu with pid 7993
I20260927 14:52:03.409945 8457 ts_tablet_manager.cc:1905] T 5bc2c777a3184f8caeeb5be6077c5dd0 P b9684a38ff254c1287b71a241d319172: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 14:52:03.409951 8455 ts_tablet_manager.cc:1905] T 5bc2c777a3184f8caeeb5be6077c5dd0 P fd36717bc96647548d91ea87dd09dae4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 14:52:03.412276 8457 ts_tablet_manager.cc:1918] T 5bc2c777a3184f8caeeb5be6077c5dd0 P b9684a38ff254c1287b71a241d319172: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 14:52:03.412276 8455 ts_tablet_manager.cc:1918] T 5bc2c777a3184f8caeeb5be6077c5dd0 P fd36717bc96647548d91ea87dd09dae4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 14:52:03.412358 8455 log.cc:1196] T 5bc2c777a3184f8caeeb5be6077c5dd0 P fd36717bc96647548d91ea87dd09dae4: Deleting WAL directory at /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790520684602251-392-0/minicluster-data/ts-2/wal/wals/5bc2c777a3184f8caeeb5be6077c5dd0
I20260927 14:52:03.412358 8457 log.cc:1196] T 5bc2c777a3184f8caeeb5be6077c5dd0 P b9684a38ff254c1287b71a241d319172: Deleting WAL directory at /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790520684602251-392-0/minicluster-data/ts-1/wal/wals/5bc2c777a3184f8caeeb5be6077c5dd0
I20260927 14:52:03.412618 8455 ts_tablet_manager.cc:1939] T 5bc2c777a3184f8caeeb5be6077c5dd0 P fd36717bc96647548d91ea87dd09dae4: Deleting consensus metadata
I20260927 14:52:03.412618 8457 ts_tablet_manager.cc:1939] T 5bc2c777a3184f8caeeb5be6077c5dd0 P b9684a38ff254c1287b71a241d319172: Deleting consensus metadata
I20260927 14:52:03.413311 7908 catalog_manager.cc:4841] TS b9684a38ff254c1287b71a241d319172 (127.0.98.2:34999): tablet 5bc2c777a3184f8caeeb5be6077c5dd0 (table test-table [id=a8169a72411f4577a4a269078a019c36]) successfully deleted
I20260927 14:52:03.414690 7905 catalog_manager.cc:4841] TS fd36717bc96647548d91ea87dd09dae4 (127.0.98.3:43169): tablet 5bc2c777a3184f8caeeb5be6077c5dd0 (table test-table [id=a8169a72411f4577a4a269078a019c36]) successfully deleted
W20260927 14:52:03.420308 7905 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:46477 (error 108)
I20260927 14:52:03.420501 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6936m5/build/debug/bin/kudu with pid 8133
I20260927 14:52:03.422477 8512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:52:03.422103 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:46477 (local address 127.0.0.1:56638)
0927 14:52:03.422334 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0927 14:52:03.422398 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:46477: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":61,"threads_started":1}
W20260927 14:52:03.422657 7905 catalog_manager.cc:4584] TS 7a267c5c521e4912aad1c6124b1a92b6 (127.0.98.1:46477): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 5bc2c777a3184f8caeeb5be6077c5dd0: Network error: Client connection negotiation failed: client connection to 127.0.98.1:46477: connect: Connection refused (error 111)
I20260927 14:52:03.426254 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6936m5/build/debug/bin/kudu with pid 8273
I20260927 14:52:03.431833 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6936m5/build/debug/bin/kudu with pid 7889
2026-09-27T14:52:03Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (3307 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790520684602251-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 14:52:03 dist-test-slave-xk1g krb5kdc[8515](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 14:52:03 dist-test-slave-xk1g krb5kdc[8515](info): set up 2 sockets
Sep 27 14:52:03 dist-test-slave-xk1g krb5kdc[8515](info): commencing operation
krb5kdc: starting...
W20260927 14:52:04.461717 392 mini_kdc.cc:121] Time spent starting KDC: real 1.007s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790520684602251-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790520684602251-392-0/krb5kdc/test-user.keytab.
Sep 27 14:52:04 dist-test-slave-xk1g krb5kdc[8515](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790520724, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T14:52:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T14:52:04Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790520684602251-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790520684602251-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 14:52:04.581527 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6936m5/build/debug/bin/kudu
/tmp/dist-test-task6936m5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790520684602251-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790520684602251-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790520684602251-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790520684602251-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36369
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790520684602251-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37273
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:36369
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790520684602251-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790520684602251-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790520684602251-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 14:52:04.691457 8528 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 14:52:04.691793 8528 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 14:52:04.691872 8528 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 14:52:04.695926 8528 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 14:52:04.696041 8528 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 14:52:04.696081 8528 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 14:52:04.696136 8528 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 14:52:04.696173 8528 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 14:52:04.701478 8528 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37273
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790520684602251-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790520684602251-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:36369
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790520684602251-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36369
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790520684602251-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8528
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790520684602251-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:15:17 UTC on ccycloud.quasar-ledxfe.root.comops.site
build id 4676134
I20260927 14:52:04.702574 8528 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 14:52:04.703631 8528 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 14:52:04.711851 8533 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 14:52:04.711836 8534 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 14:52:04.712144 8528 server_base.cc:1034] running on GCE node
W20260927 14:52:04.712141 8536 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 14:52:04.712837 8528 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 14:52:04.713971 8528 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 14:52:04.715137 8528 hybrid_clock.cc:648] HybridClock initialized: now 1790520724715122 us; error 35 us; skew 500 ppm
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-09-27T14:52:04Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1286 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790520684602251-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 14:52:04 dist-test-slave-xk1g krb5kdc[8541](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 14:52:04 dist-test-slave-xk1g krb5kdc[8541](info): set up 2 sockets
Sep 27 14:52:04 dist-test-slave-xk1g krb5kdc[8541](info): commencing operation
krb5kdc: starting...
W20260927 14:52:05.761938 392 mini_kdc.cc:121] Time spent starting KDC: real 1.021s user 0.003s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790520684602251-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790520684602251-392-0/krb5kdc/test-user.keytab.
Sep 27 14:52:05 dist-test-slave-xk1g krb5kdc[8541](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790520725, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T14:52:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T14:52:05Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790520684602251-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790520684602251-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 14:52:05.879794 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6936m5/build/debug/bin/kudu
/tmp/dist-test-task6936m5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790520684602251-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790520684602251-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790520684602251-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790520684602251-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39465
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790520684602251-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33137
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:39465
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790520684602251-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790520684602251-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790520684602251-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 14:52:05.987747 8554 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 14:52:05.988024 8554 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 14:52:05.988066 8554 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 14:52:05.992076 8554 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 14:52:05.992143 8554 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 14:52:05.992163 8554 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 14:52:05.992179 8554 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 14:52:05.992195 8554 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 14:52:05.997442 8554 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33137
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790520684602251-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790520684602251-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:39465
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790520684602251-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39465
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790520684602251-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8554
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790520684602251-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:15:17 UTC on ccycloud.quasar-ledxfe.root.comops.site
build id 4676134
I20260927 14:52:05.998436 8554 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 14:52:05.999387 8554 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 14:52:06.007848 8562 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 14:52:06.007848 8560 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 14:52:06.007936 8554 server_base.cc:1034] running on GCE node
W20260927 14:52:06.007882 8559 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 14:52:06.008775 8554 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 14:52:06.009863 8554 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 14:52:06.011014 8554 hybrid_clock.cc:648] HybridClock initialized: now 1790520726010986 us; error 43 us; skew 500 ppm
Sep 27 14:52:06 dist-test-slave-xk1g krb5kdc[8541](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790520726, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 14:52:06.019747 8554 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 14:52:06.020735 8554 webserver.cc:460] Webserver started at http://127.0.98.62:39313/ using document root <none> and password file <none>
I20260927 14:52:06.021404 8554 fs_manager.cc:359] Metadata directory not provided
I20260927 14:52:06.021499 8554 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 14:52:06.021735 8554 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 14:52:06.023792 8554 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790520684602251-392-0/minicluster-data/master-0/data/instance:
uuid: "d5e64f44053d4052bf1651dfb339497d"
format_stamp: "Formatted at 2026-09-27 14:52:06 on dist-test-slave-xk1g"
server_key: "442cad59a9a8c2a5ef4554c0820382ee"
server_key_iv: "a4d2c0358a201da5946e62b95a4b4255"
server_key_version: "encryptionkey@0"
I20260927 14:52:06.024358 8554 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790520684602251-392-0/minicluster-data/master-0/wal/instance:
uuid: "d5e64f44053d4052bf1651dfb339497d"
format_stamp: "Formatted at 2026-09-27 14:52:06 on dist-test-slave-xk1g"
server_key: "442cad59a9a8c2a5ef4554c0820382ee"
server_key_iv: "a4d2c0358a201da5946e62b95a4b4255"
server_key_version: "encryptionkey@0"
I20260927 14:52:06.028064 8554 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260927 14:52:06.030529 8569 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 14:52:06.031891 8554 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 14:52:06.032054 8554 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790520684602251-392-0/minicluster-data/master-0/data,/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790520684602251-392-0/minicluster-data/master-0/wal
uuid: "d5e64f44053d4052bf1651dfb339497d"
format_stamp: "Formatted at 2026-09-27 14:52:06 on dist-test-slave-xk1g"
server_key: "442cad59a9a8c2a5ef4554c0820382ee"
server_key_iv: "a4d2c0358a201da5946e62b95a4b4255"
server_key_version: "encryptionkey@0"
I20260927 14:52:06.032176 8554 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790520684602251-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790520684602251-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790520684602251-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 14:52:06.066736 8554 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 14:52:06.070510 8554 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 14:52:06.070757 8554 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 14:52:06.078521 8554 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:39465
I20260927 14:52:06.078534 8621 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:39465 every 8 connection(s)
I20260927 14:52:06.079514 8554 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790520684602251-392-0/minicluster-data/master-0/data/info.pb
I20260927 14:52:06.082384 8622 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 14:52:06.085994 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6936m5/build/debug/bin/kudu as pid 8554
I20260927 14:52:06.086126 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790520684602251-392-0/minicluster-data/master-0/wal/instance
I20260927 14:52:06.086421 392 external_mini_cluster.cc:1428] Setting key 6e0687738382e88fc56f7eeaa829a8c4
I20260927 14:52:06.088575 8622 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d5e64f44053d4052bf1651dfb339497d: Bootstrap starting.
I20260927 14:52:06.090824 8622 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d5e64f44053d4052bf1651dfb339497d: Neither blocks nor log segments found. Creating new log.
I20260927 14:52:06.091542 8622 log.cc:824] T 00000000000000000000000000000000 P d5e64f44053d4052bf1651dfb339497d: Log is configured to *not* fsync() on all Append() calls
I20260927 14:52:06.093492 8622 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d5e64f44053d4052bf1651dfb339497d: No bootstrap required, opened a new log
Sep 27 14:52:06 dist-test-slave-xk1g krb5kdc[8541](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790520725, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 14:52:06.097932 8622 raft_consensus.cc:348] T 00000000000000000000000000000000 P d5e64f44053d4052bf1651dfb339497d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d5e64f44053d4052bf1651dfb339497d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39465 } }
I20260927 14:52:06.098224 8622 raft_consensus.cc:374] T 00000000000000000000000000000000 P d5e64f44053d4052bf1651dfb339497d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 14:52:06.098304 8622 raft_consensus.cc:729] T 00000000000000000000000000000000 P d5e64f44053d4052bf1651dfb339497d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d5e64f44053d4052bf1651dfb339497d, State: Initialized, Role: FOLLOWER
I20260927 14:52:06.098819 8622 consensus_queue.cc:260] T 00000000000000000000000000000000 P d5e64f44053d4052bf1651dfb339497d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d5e64f44053d4052bf1651dfb339497d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39465 } }
I20260927 14:52:06.098984 8622 raft_consensus.cc:388] T 00000000000000000000000000000000 P d5e64f44053d4052bf1651dfb339497d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 14:52:06.099066 8622 raft_consensus.cc:482] T 00000000000000000000000000000000 P d5e64f44053d4052bf1651dfb339497d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 14:52:06.099174 8622 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d5e64f44053d4052bf1651dfb339497d [term 0 FOLLOWER]: Advancing to term 1
I20260927 14:52:06.100323 8622 raft_consensus.cc:504] T 00000000000000000000000000000000 P d5e64f44053d4052bf1651dfb339497d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d5e64f44053d4052bf1651dfb339497d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39465 } }
I20260927 14:52:06.100668 8622 leader_election.cc:302] T 00000000000000000000000000000000 P d5e64f44053d4052bf1651dfb339497d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d5e64f44053d4052bf1651dfb339497d; no voters:
I20260927 14:52:06.100994 8622 leader_election.cc:288] T 00000000000000000000000000000000 P d5e64f44053d4052bf1651dfb339497d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 14:52:06.101126 8627 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d5e64f44053d4052bf1651dfb339497d [term 1 FOLLOWER]: Leader election won for term 1
I20260927 14:52:06.101370 8627 raft_consensus.cc:686] T 00000000000000000000000000000000 P d5e64f44053d4052bf1651dfb339497d [term 1 LEADER]: Becoming Leader. State: Replica: d5e64f44053d4052bf1651dfb339497d, State: Running, Role: LEADER
I20260927 14:52:06.101547 8625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:52:06.087875 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33484 (local address 127.0.98.62:39465)
0927 14:52:06.088326 (+ 451us) server_negotiation.cc:206] Beginning negotiation
0927 14:52:06.088341 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 14:52:06.088378 (+ 37us) server_negotiation.cc:407] Connection header received
0927 14:52:06.089118 (+ 740us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:52:06.089150 (+ 32us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:52:06.089514 (+ 364us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:52:06.089844 (+ 330us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:52:06.090651 (+ 807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:52:06.091718 (+ 1067us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:52:06.092716 (+ 998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:52:06.093251 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:52:06.093434 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:52:06.096144 (+ 2710us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:52:06.096153 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:52:06.096173 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 14:52:06.096217 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:52:06.098250 (+ 2033us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:52:06.098778 (+ 528us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:52:06.098784 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:52:06.098789 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:52:06.098932 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:52:06.099248 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:52:06.099252 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:52:06.099254 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:52:06.099784 (+ 530us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:52:06.099975 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0927 14:52:06.100214 (+ 239us) server_negotiation.cc:299] Negotiation successful
0927 14:52:06.100520 (+ 306us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":300,"thread_start_us":132,"threads_started":1}
I20260927 14:52:06.101727 8627 consensus_queue.cc:237] T 00000000000000000000000000000000 P d5e64f44053d4052bf1651dfb339497d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d5e64f44053d4052bf1651dfb339497d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39465 } }
I20260927 14:52:06.102226 8622 sys_catalog.cc:564] T 00000000000000000000000000000000 P d5e64f44053d4052bf1651dfb339497d [sys.catalog]: configured and running, proceeding with master startup.
I20260927 14:52:06.102864 8627 sys_catalog.cc:455] T 00000000000000000000000000000000 P d5e64f44053d4052bf1651dfb339497d [sys.catalog]: SysCatalogTable state changed. Reason: New leader d5e64f44053d4052bf1651dfb339497d. Latest consensus state: current_term: 1 leader_uuid: "d5e64f44053d4052bf1651dfb339497d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d5e64f44053d4052bf1651dfb339497d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39465 } } }
I20260927 14:52:06.103015 8627 sys_catalog.cc:458] T 00000000000000000000000000000000 P d5e64f44053d4052bf1651dfb339497d [sys.catalog]: This master's current role is: LEADER
I20260927 14:52:06.103292 8629 sys_catalog.cc:455] T 00000000000000000000000000000000 P d5e64f44053d4052bf1651dfb339497d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d5e64f44053d4052bf1651dfb339497d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d5e64f44053d4052bf1651dfb339497d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39465 } } }
I20260927 14:52:06.103425 8629 sys_catalog.cc:458] T 00000000000000000000000000000000 P d5e64f44053d4052bf1651dfb339497d [sys.catalog]: This master's current role is: LEADER
I20260927 14:52:06.104534 8634 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 14:52:06.107290 8634 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 14:52:06.112473 8634 catalog_manager.cc:1347] Generated new cluster ID: f38914fb598340d58f9b35a5735e512f
I20260927 14:52:06.112568 8634 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 14:52:06.127055 8634 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 14:52:06.128002 8634 catalog_manager.cc:1504] Loading token signing keys...
I20260927 14:52:06.133646 8634 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d5e64f44053d4052bf1651dfb339497d: Generated new TSK 0
I20260927 14:52:06.134497 8634 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790520684602251-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790520684602251-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260927 14:52:06.168555 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6936m5/build/debug/bin/kudu
/tmp/dist-test-task6936m5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790520684602251-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790520684602251-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790520684602251-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790520684602251-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39465
--keytab_file=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790520684602251-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33137
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790520684602251-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790520684602251-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790520684602251-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 14:52:06.277235 8648 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 14:52:06.277491 8648 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 14:52:06.277534 8648 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 14:52:06.281545 8648 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 14:52:06.281598 8648 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 14:52:06.281690 8648 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 14:52:06.286890 8648 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33137
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790520684602251-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790520684602251-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790520684602251-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790520684602251-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39465
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8648
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790520684602251-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:15:17 UTC on ccycloud.quasar-ledxfe.root.comops.site
build id 4676134
I20260927 14:52:06.287922 8648 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 14:52:06.288971 8648 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 14:52:06.298324 8653 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 14:52:06.298487 8656 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 14:52:06.298365 8648 server_base.cc:1034] running on GCE node
W20260927 14:52:06.298334 8654 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 14:52:06.298949 8648 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 14:52:06.299480 8648 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 14:52:06.300633 8648 hybrid_clock.cc:648] HybridClock initialized: now 1790520726300622 us; error 25 us; skew 500 ppm
Sep 27 14:52:06 dist-test-slave-xk1g krb5kdc[8541](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790520726, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 14:52:06.303462 8648 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 14:52:06.304253 8648 webserver.cc:460] Webserver started at http://127.0.98.1:43299/ using document root <none> and password file <none>
I20260927 14:52:06.304780 8648 fs_manager.cc:359] Metadata directory not provided
I20260927 14:52:06.304869 8648 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 14:52:06.305158 8648 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 14:52:06.307087 8648 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790520684602251-392-0/minicluster-data/ts-0/data/instance:
uuid: "fc33b763315b403784837872a2ce63a9"
format_stamp: "Formatted at 2026-09-27 14:52:06 on dist-test-slave-xk1g"
server_key: "cc729632985d97e3765d9a53a0aae2b5"
server_key_iv: "2a5986fa665f1635d591854fdfbe4409"
server_key_version: "encryptionkey@0"
I20260927 14:52:06.307618 8648 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790520684602251-392-0/minicluster-data/ts-0/wal/instance:
uuid: "fc33b763315b403784837872a2ce63a9"
format_stamp: "Formatted at 2026-09-27 14:52:06 on dist-test-slave-xk1g"
server_key: "cc729632985d97e3765d9a53a0aae2b5"
server_key_iv: "2a5986fa665f1635d591854fdfbe4409"
server_key_version: "encryptionkey@0"
I20260927 14:52:06.311075 8648 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260927 14:52:06.313387 8663 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 14:52:06.314558 8648 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260927 14:52:06.314685 8648 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790520684602251-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790520684602251-392-0/minicluster-data/ts-0/wal
uuid: "fc33b763315b403784837872a2ce63a9"
format_stamp: "Formatted at 2026-09-27 14:52:06 on dist-test-slave-xk1g"
server_key: "cc729632985d97e3765d9a53a0aae2b5"
server_key_iv: "2a5986fa665f1635d591854fdfbe4409"
server_key_version: "encryptionkey@0"
I20260927 14:52:06.314791 8648 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790520684602251-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790520684602251-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790520684602251-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 14:52:06.335801 8648 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 14:52:06.339145 8648 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 14:52:06.339351 8648 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 14:52:06.339947 8648 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 14:52:06.340751 8648 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 14:52:06.340821 8648 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 14:52:06.340895 8648 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 14:52:06.340946 8648 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 14:52:06.350033 8648 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:36023
I20260927 14:52:06.350062 8776 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:36023 every 8 connection(s)
I20260927 14:52:06.350917 8648 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790520684602251-392-0/minicluster-data/ts-0/data/info.pb
I20260927 14:52:06.354920 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6936m5/build/debug/bin/kudu as pid 8648
I20260927 14:52:06.355054 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790520684602251-392-0/minicluster-data/ts-0/wal/instance
I20260927 14:52:06.355326 392 external_mini_cluster.cc:1428] Setting key e658bc18b277bdc95c77b0798a80c89f
Sep 27 14:52:06 dist-test-slave-xk1g krb5kdc[8541](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790520726, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260927 14:52:06.365973 8625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:52:06.352751 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:53851 (local address 127.0.98.62:39465)
0927 14:52:06.352884 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0927 14:52:06.352888 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 14:52:06.353719 (+ 831us) server_negotiation.cc:407] Connection header received
0927 14:52:06.355101 (+ 1382us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:52:06.355104 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:52:06.355164 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:52:06.355241 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:52:06.356750 (+ 1509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:52:06.357543 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:52:06.359015 (+ 1472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:52:06.359233 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:52:06.359315 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:52:06.362272 (+ 2957us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:52:06.362274 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:52:06.362276 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 14:52:06.362299 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:52:06.363749 (+ 1450us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:52:06.364389 (+ 640us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:52:06.364391 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:52:06.364393 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:52:06.364470 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:52:06.364863 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:52:06.364869 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:52:06.364870 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:52:06.365096 (+ 226us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:52:06.365177 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0927 14:52:06.365663 (+ 486us) server_negotiation.cc:299] Negotiation successful
0927 14:52:06.365761 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260927 14:52:06.366758 8779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:52:06.353158 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39465 (local address 127.0.98.1:53851)
0927 14:52:06.353571 (+ 413us) negotiation.cc:107] Waiting for socket to connect
0927 14:52:06.353612 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0927 14:52:06.354874 (+ 1262us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:52:06.355409 (+ 535us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:52:06.355438 (+ 29us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:52:06.355905 (+ 467us) client_negotiation.cc:189] Negotiated authn=SASL
0927 14:52:06.356545 (+ 640us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:52:06.356561 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:52:06.357661 (+ 1100us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:52:06.357664 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:52:06.358873 (+ 1209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:52:06.358887 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:52:06.359384 (+ 497us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:52:06.359388 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:52:06.359610 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:52:06.360208 (+ 598us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 14:52:06.360223 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 14:52:06.362095 (+ 1872us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 14:52:06.363882 (+ 1787us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 14:52:06.363890 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 14:52:06.363911 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0927 14:52:06.364286 (+ 375us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 14:52:06.364563 (+ 277us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 14:52:06.364566 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 14:52:06.364568 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 14:52:06.364740 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 14:52:06.365206 (+ 466us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 14:52:06.365220 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 14:52:06.365457 (+ 237us) client_negotiation.cc:769] Sending connection context
0927 14:52:06.365621 (+ 164us) client_negotiation.cc:240] Negotiation successful
0927 14:52:06.365923 (+ 302us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":236,"thread_start_us":100,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260927 14:52:06.367983 8777 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39465
I20260927 14:52:06.368232 8777 heartbeater.cc:464] Registering TS with master...
I20260927 14:52:06.368744 8777 heartbeater.cc:511] Master 127.0.98.62:39465 requested a full tablet report, sending...
I20260927 14:52:06.370393 8586 ts_manager.cc:194] Registered new tserver with Master: fc33b763315b403784837872a2ce63a9 (127.0.98.1:36023)
I20260927 14:52:06.371461 8586 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:53851
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790520684602251-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790520684602251-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260927 14:52:06.383857 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6936m5/build/debug/bin/kudu
/tmp/dist-test-task6936m5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790520684602251-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790520684602251-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790520684602251-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790520684602251-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39465
--keytab_file=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790520684602251-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33137
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790520684602251-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790520684602251-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790520684602251-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 14:52:06.494911 8782 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 14:52:06.495164 8782 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 14:52:06.495208 8782 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 14:52:06.499361 8782 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 14:52:06.499419 8782 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 14:52:06.499480 8782 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 14:52:06.505028 8782 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33137
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790520684602251-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790520684602251-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790520684602251-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790520684602251-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39465
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8782
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790520684602251-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:15:17 UTC on ccycloud.quasar-ledxfe.root.comops.site
build id 4676134
I20260927 14:52:06.506098 8782 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 14:52:06.507073 8782 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 14:52:06.518150 8790 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 14:52:06.518270 8788 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 14:52:06.518461 8782 server_base.cc:1034] running on GCE node
W20260927 14:52:06.518316 8787 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 14:52:06.519155 8782 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 14:52:06.519950 8782 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 14:52:06.521132 8782 hybrid_clock.cc:648] HybridClock initialized: now 1790520726521103 us; error 44 us; skew 500 ppm
Sep 27 14:52:06 dist-test-slave-xk1g krb5kdc[8541](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790520726, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 14:52:06.523989 8782 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 14:52:06.524825 8782 webserver.cc:460] Webserver started at http://127.0.98.2:37783/ using document root <none> and password file <none>
I20260927 14:52:06.525449 8782 fs_manager.cc:359] Metadata directory not provided
I20260927 14:52:06.525542 8782 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 14:52:06.525767 8782 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 14:52:06.527809 8782 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790520684602251-392-0/minicluster-data/ts-1/data/instance:
uuid: "afc62c555d7f44c0aee158e1d96ee9e7"
format_stamp: "Formatted at 2026-09-27 14:52:06 on dist-test-slave-xk1g"
server_key: "5c7feac7aa4d0a9750d418b176694780"
server_key_iv: "96401e5ab866647f5f12e8e9da901ccc"
server_key_version: "encryptionkey@0"
I20260927 14:52:06.528373 8782 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790520684602251-392-0/minicluster-data/ts-1/wal/instance:
uuid: "afc62c555d7f44c0aee158e1d96ee9e7"
format_stamp: "Formatted at 2026-09-27 14:52:06 on dist-test-slave-xk1g"
server_key: "5c7feac7aa4d0a9750d418b176694780"
server_key_iv: "96401e5ab866647f5f12e8e9da901ccc"
server_key_version: "encryptionkey@0"
I20260927 14:52:06.532053 8782 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260927 14:52:06.534572 8797 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 14:52:06.535830 8782 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 14:52:06.535974 8782 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790520684602251-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790520684602251-392-0/minicluster-data/ts-1/wal
uuid: "afc62c555d7f44c0aee158e1d96ee9e7"
format_stamp: "Formatted at 2026-09-27 14:52:06 on dist-test-slave-xk1g"
server_key: "5c7feac7aa4d0a9750d418b176694780"
server_key_iv: "96401e5ab866647f5f12e8e9da901ccc"
server_key_version: "encryptionkey@0"
I20260927 14:52:06.536103 8782 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790520684602251-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790520684602251-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790520684602251-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 14:52:06.552528 8782 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 14:52:06.555961 8782 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 14:52:06.556162 8782 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 14:52:06.556748 8782 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 14:52:06.557615 8782 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 14:52:06.557685 8782 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 14:52:06.557755 8782 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 14:52:06.557806 8782 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 14:52:06.566027 8782 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:45779
I20260927 14:52:06.566082 8910 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:45779 every 8 connection(s)
I20260927 14:52:06.566921 8782 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790520684602251-392-0/minicluster-data/ts-1/data/info.pb
I20260927 14:52:06.570276 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6936m5/build/debug/bin/kudu as pid 8782
I20260927 14:52:06.570426 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790520684602251-392-0/minicluster-data/ts-1/wal/instance
I20260927 14:52:06.570714 392 external_mini_cluster.cc:1428] Setting key 7655c0ed806720bd7afe329b5c436daa
Sep 27 14:52:06 dist-test-slave-xk1g krb5kdc[8541](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790520726, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260927 14:52:06.581418 8625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:52:06.568716 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:38319 (local address 127.0.98.62:39465)
0927 14:52:06.568850 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0927 14:52:06.568854 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 14:52:06.569780 (+ 926us) server_negotiation.cc:407] Connection header received
0927 14:52:06.571256 (+ 1476us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:52:06.571259 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:52:06.571315 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:52:06.571422 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:52:06.572552 (+ 1130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:52:06.573124 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:52:06.574452 (+ 1328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:52:06.574859 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:52:06.574992 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:52:06.577548 (+ 2556us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:52:06.577555 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:52:06.577559 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 14:52:06.577592 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:52:06.579284 (+ 1692us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:52:06.579793 (+ 509us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:52:06.579796 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:52:06.579798 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:52:06.579882 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:52:06.580258 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:52:06.580265 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:52:06.580270 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:52:06.580480 (+ 210us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:52:06.580561 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0927 14:52:06.581173 (+ 612us) server_negotiation.cc:299] Negotiation successful
0927 14:52:06.581287 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260927 14:52:06.582370 8913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:52:06.569191 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39465 (local address 127.0.98.2:38319)
0927 14:52:06.569618 (+ 427us) negotiation.cc:107] Waiting for socket to connect
0927 14:52:06.569660 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0927 14:52:06.571043 (+ 1383us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:52:06.571600 (+ 557us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:52:06.571607 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:52:06.571955 (+ 348us) client_negotiation.cc:189] Negotiated authn=SASL
0927 14:52:06.572416 (+ 461us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:52:06.572429 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:52:06.573262 (+ 833us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:52:06.573269 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:52:06.574326 (+ 1057us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:52:06.574336 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:52:06.575012 (+ 676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:52:06.575014 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:52:06.575162 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:52:06.575826 (+ 664us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 14:52:06.575846 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0927 14:52:06.577368 (+ 1522us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 14:52:06.579414 (+ 2046us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 14:52:06.579419 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 14:52:06.579432 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0927 14:52:06.579691 (+ 259us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 14:52:06.579984 (+ 293us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 14:52:06.579986 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 14:52:06.579987 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 14:52:06.580163 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 14:52:06.580603 (+ 440us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 14:52:06.580608 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 14:52:06.580861 (+ 253us) client_negotiation.cc:769] Sending connection context
0927 14:52:06.581105 (+ 244us) client_negotiation.cc:240] Negotiation successful
0927 14:52:06.581411 (+ 306us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":251,"thread_start_us":107,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260927 14:52:06.583726 8911 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39465
I20260927 14:52:06.583983 8911 heartbeater.cc:464] Registering TS with master...
I20260927 14:52:06.584547 8911 heartbeater.cc:511] Master 127.0.98.62:39465 requested a full tablet report, sending...
I20260927 14:52:06.585774 8586 ts_manager.cc:194] Registered new tserver with Master: afc62c555d7f44c0aee158e1d96ee9e7 (127.0.98.2:45779)
I20260927 14:52:06.586369 8586 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:38319
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790520684602251-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790520684602251-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260927 14:52:06.599939 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6936m5/build/debug/bin/kudu
/tmp/dist-test-task6936m5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790520684602251-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790520684602251-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790520684602251-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790520684602251-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39465
--keytab_file=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790520684602251-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33137
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790520684602251-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790520684602251-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790520684602251-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 14:52:06.709477 8916 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 14:52:06.709738 8916 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 14:52:06.709779 8916 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 14:52:06.714026 8916 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 14:52:06.714076 8916 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 14:52:06.714178 8916 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 14:52:06.719679 8916 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33137
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790520684602251-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790520684602251-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790520684602251-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790520684602251-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39465
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8916
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790520684602251-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:15:17 UTC on ccycloud.quasar-ledxfe.root.comops.site
build id 4676134
I20260927 14:52:06.720718 8916 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 14:52:06.721766 8916 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 14:52:06.731226 8922 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 14:52:06.731245 8924 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 14:52:06.731336 8916 server_base.cc:1034] running on GCE node
W20260927 14:52:06.731249 8921 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 14:52:06.731801 8916 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 14:52:06.732410 8916 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 14:52:06.733592 8916 hybrid_clock.cc:648] HybridClock initialized: now 1790520726733579 us; error 28 us; skew 500 ppm
Sep 27 14:52:06 dist-test-slave-xk1g krb5kdc[8541](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790520726, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 14:52:06.736222 8916 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 14:52:06.737089 8916 webserver.cc:460] Webserver started at http://127.0.98.3:33569/ using document root <none> and password file <none>
I20260927 14:52:06.737603 8916 fs_manager.cc:359] Metadata directory not provided
I20260927 14:52:06.737660 8916 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 14:52:06.737814 8916 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 14:52:06.739681 8916 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790520684602251-392-0/minicluster-data/ts-2/data/instance:
uuid: "d40da6e81dac4fb4ac5bc8ebd9208645"
format_stamp: "Formatted at 2026-09-27 14:52:06 on dist-test-slave-xk1g"
server_key: "c126f60b7b1533e2c787a916e715f1cd"
server_key_iv: "f95fdc71843c79d9f38f483149d41fbb"
server_key_version: "encryptionkey@0"
I20260927 14:52:06.740198 8916 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790520684602251-392-0/minicluster-data/ts-2/wal/instance:
uuid: "d40da6e81dac4fb4ac5bc8ebd9208645"
format_stamp: "Formatted at 2026-09-27 14:52:06 on dist-test-slave-xk1g"
server_key: "c126f60b7b1533e2c787a916e715f1cd"
server_key_iv: "f95fdc71843c79d9f38f483149d41fbb"
server_key_version: "encryptionkey@0"
I20260927 14:52:06.743719 8916 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260927 14:52:06.745946 8931 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 14:52:06.747134 8916 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 14:52:06.747221 8916 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790520684602251-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790520684602251-392-0/minicluster-data/ts-2/wal
uuid: "d40da6e81dac4fb4ac5bc8ebd9208645"
format_stamp: "Formatted at 2026-09-27 14:52:06 on dist-test-slave-xk1g"
server_key: "c126f60b7b1533e2c787a916e715f1cd"
server_key_iv: "f95fdc71843c79d9f38f483149d41fbb"
server_key_version: "encryptionkey@0"
I20260927 14:52:06.747298 8916 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790520684602251-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790520684602251-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790520684602251-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 14:52:06.767074 8916 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 14:52:06.769932 8916 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 14:52:06.770079 8916 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 14:52:06.770578 8916 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 14:52:06.771322 8916 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 14:52:06.771366 8916 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 14:52:06.771405 8916 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 14:52:06.771426 8916 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 14:52:06.780085 8916 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:44407
I20260927 14:52:06.780110 9044 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:44407 every 8 connection(s)
I20260927 14:52:06.780931 8916 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790520684602251-392-0/minicluster-data/ts-2/data/info.pb
I20260927 14:52:06.786430 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6936m5/build/debug/bin/kudu as pid 8916
I20260927 14:52:06.786542 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790520684602251-392-0/minicluster-data/ts-2/wal/instance
I20260927 14:52:06.786800 392 external_mini_cluster.cc:1428] Setting key eb0cdc21513f19c8edad833ccd3fdbe7
Sep 27 14:52:06 dist-test-slave-xk1g krb5kdc[8541](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790520726, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 14:52:06.794093 8625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:52:06.782684 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:42285 (local address 127.0.98.62:39465)
0927 14:52:06.782846 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0927 14:52:06.782851 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 14:52:06.783663 (+ 812us) server_negotiation.cc:407] Connection header received
0927 14:52:06.784943 (+ 1280us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:52:06.784945 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:52:06.784989 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:52:06.785121 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:52:06.786274 (+ 1153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:52:06.787017 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:52:06.788360 (+ 1343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:52:06.788655 (+ 295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:52:06.788760 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:52:06.790933 (+ 2173us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:52:06.790937 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:52:06.790939 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 14:52:06.790981 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:52:06.792151 (+ 1170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:52:06.792617 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:52:06.792622 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:52:06.792623 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:52:06.792705 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:52:06.793039 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:52:06.793044 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:52:06.793046 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:52:06.793246 (+ 200us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:52:06.793353 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0927 14:52:06.793803 (+ 450us) server_negotiation.cc:299] Negotiation successful
0927 14:52:06.793916 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260927 14:52:06.794814 9047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:52:06.783100 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39465 (local address 127.0.98.3:42285)
0927 14:52:06.783514 (+ 414us) negotiation.cc:107] Waiting for socket to connect
0927 14:52:06.783556 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0927 14:52:06.784748 (+ 1192us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:52:06.785301 (+ 553us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:52:06.785307 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:52:06.785643 (+ 336us) client_negotiation.cc:189] Negotiated authn=SASL
0927 14:52:06.786126 (+ 483us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:52:06.786139 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:52:06.787124 (+ 985us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:52:06.787128 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:52:06.788243 (+ 1115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:52:06.788251 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:52:06.788792 (+ 541us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:52:06.788796 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:52:06.788968 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:52:06.789475 (+ 507us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 14:52:06.789489 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0927 14:52:06.790786 (+ 1297us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 14:52:06.792259 (+ 1473us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 14:52:06.792265 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 14:52:06.792278 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0927 14:52:06.792514 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 14:52:06.792805 (+ 291us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 14:52:06.792808 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 14:52:06.792810 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 14:52:06.792944 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 14:52:06.793348 (+ 404us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 14:52:06.793354 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 14:52:06.793582 (+ 228us) client_negotiation.cc:769] Sending connection context
0927 14:52:06.793753 (+ 171us) client_negotiation.cc:240] Negotiation successful
0927 14:52:06.794067 (+ 314us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":227,"thread_start_us":96,"threads_started":1}
I20260927 14:52:06.795794 9045 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39465
I20260927 14:52:06.796023 9045 heartbeater.cc:464] Registering TS with master...
I20260927 14:52:06.796455 9045 heartbeater.cc:511] Master 127.0.98.62:39465 requested a full tablet report, sending...
I20260927 14:52:06.797405 8586 ts_manager.cc:194] Registered new tserver with Master: d40da6e81dac4fb4ac5bc8ebd9208645 (127.0.98.3:44407)
I20260927 14:52:06.797920 8586 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:42285
I20260927 14:52:06.800153 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790520684602251-392-0/krb5kdc/oryx_127.0.98.1.keytab.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790520684602251-392-0/krb5kdc/oryx_127.0.98.1.keytab.
WARNING: no policy specified for oryx/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790520684602251-392-0/krb5kdc/oryx_127.0.98.2.keytab.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790520684602251-392-0/krb5kdc/oryx_127.0.98.2.keytab.
WARNING: no policy specified for oryx/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790520684602251-392-0/krb5kdc/oryx_127.0.98.3.keytab.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790520684602251-392-0/krb5kdc/oryx_127.0.98.3.keytab.
I20260927 14:52:06.867115 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6936m5/build/debug/bin/kudu with pid 8648
I20260927 14:52:06.872438 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6936m5/build/debug/bin/kudu with pid 8782
I20260927 14:52:06.877453 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6936m5/build/debug/bin/kudu with pid 8916
I20260927 14:52:06.882503 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6936m5/build/debug/bin/kudu with pid 8554
I20260927 14:52:06.888477 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6936m5/build/debug/bin/kudu
/tmp/dist-test-task6936m5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790520684602251-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790520684602251-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790520684602251-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790520684602251-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39465
--webserver_interface=127.0.98.62
--webserver_port=39313
--keytab_file=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790520684602251-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33137
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:39465
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790520684602251-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790520684602251-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790520684602251-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 14:52:06.996829 9054 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 14:52:06.997177 9054 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 14:52:06.997268 9054 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 14:52:07.001284 9054 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 14:52:07.001386 9054 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 14:52:07.001423 9054 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 14:52:07.001461 9054 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 14:52:07.001492 9054 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 14:52:07.006831 9054 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33137
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790520684602251-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790520684602251-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:39465
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790520684602251-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39465
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790520684602251-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=39313
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9054
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790520684602251-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:15:17 UTC on ccycloud.quasar-ledxfe.root.comops.site
build id 4676134
I20260927 14:52:07.007961 9054 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 14:52:07.009006 9054 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 14:52:07.017602 9060 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 14:52:07.017624 9062 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 14:52:07.017841 9054 server_base.cc:1034] running on GCE node
W20260927 14:52:07.017972 9059 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 14:52:07.018575 9054 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 14:52:07.019726 9054 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 14:52:07.020948 9054 hybrid_clock.cc:648] HybridClock initialized: now 1790520727020936 us; error 33 us; skew 500 ppm
Sep 27 14:52:07 dist-test-slave-xk1g krb5kdc[8541](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790520727, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 14:52:07.023700 9054 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 14:52:07.024696 9054 webserver.cc:460] Webserver started at http://127.0.98.62:39313/ using document root <none> and password file <none>
I20260927 14:52:07.025341 9054 fs_manager.cc:359] Metadata directory not provided
I20260927 14:52:07.025446 9054 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 14:52:07.029310 9054 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.004s sys 0.000s
I20260927 14:52:07.031420 9069 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 14:52:07.032693 9054 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 14:52:07.032856 9054 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790520684602251-392-0/minicluster-data/master-0/data,/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790520684602251-392-0/minicluster-data/master-0/wal
uuid: "d5e64f44053d4052bf1651dfb339497d"
format_stamp: "Formatted at 2026-09-27 14:52:06 on dist-test-slave-xk1g"
server_key: "442cad59a9a8c2a5ef4554c0820382ee"
server_key_iv: "a4d2c0358a201da5946e62b95a4b4255"
server_key_version: "encryptionkey@0"
I20260927 14:52:07.033481 9054 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790520684602251-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790520684602251-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790520684602251-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 14:52:07.055289 9054 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 14:52:07.058020 9054 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 14:52:07.058221 9054 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 14:52:07.065629 9054 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:39465
I20260927 14:52:07.065647 9121 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:39465 every 8 connection(s)
I20260927 14:52:07.066684 9054 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790520684602251-392-0/minicluster-data/master-0/data/info.pb
I20260927 14:52:07.070731 9122 sys_catalog.cc:263] Verifying existing consensus state
I20260927 14:52:07.072317 9122 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d5e64f44053d4052bf1651dfb339497d: Bootstrap starting.
I20260927 14:52:07.074579 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6936m5/build/debug/bin/kudu as pid 9054
I20260927 14:52:07.075057 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6936m5/build/debug/bin/kudu
/tmp/dist-test-task6936m5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790520684602251-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790520684602251-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790520684602251-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790520684602251-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:36023
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:39465
--webserver_port=43299
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790520684602251-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33137
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.98.1
--keytab_file=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790520684602251-392-0/krb5kdc/oryx_127.0.98.1.keytab with env {KRB5CCNAME=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790520684602251-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790520684602251-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790520684602251-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 14:52:07.089001 9122 log.cc:824] T 00000000000000000000000000000000 P d5e64f44053d4052bf1651dfb339497d: Log is configured to *not* fsync() on all Append() calls
I20260927 14:52:07.095311 9122 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d5e64f44053d4052bf1651dfb339497d: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 14:52:07.095671 9122 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d5e64f44053d4052bf1651dfb339497d: Bootstrap complete.
I20260927 14:52:07.099884 9122 raft_consensus.cc:348] T 00000000000000000000000000000000 P d5e64f44053d4052bf1651dfb339497d [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d5e64f44053d4052bf1651dfb339497d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39465 } }
I20260927 14:52:07.100270 9122 raft_consensus.cc:729] T 00000000000000000000000000000000 P d5e64f44053d4052bf1651dfb339497d [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: d5e64f44053d4052bf1651dfb339497d, State: Initialized, Role: FOLLOWER
I20260927 14:52:07.100781 9122 consensus_queue.cc:260] T 00000000000000000000000000000000 P d5e64f44053d4052bf1651dfb339497d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d5e64f44053d4052bf1651dfb339497d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39465 } }
I20260927 14:52:07.100895 9122 raft_consensus.cc:388] T 00000000000000000000000000000000 P d5e64f44053d4052bf1651dfb339497d [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 14:52:07.100986 9122 raft_consensus.cc:482] T 00000000000000000000000000000000 P d5e64f44053d4052bf1651dfb339497d [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 14:52:07.101116 9122 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d5e64f44053d4052bf1651dfb339497d [term 1 FOLLOWER]: Advancing to term 2
I20260927 14:52:07.102962 9122 raft_consensus.cc:504] T 00000000000000000000000000000000 P d5e64f44053d4052bf1651dfb339497d [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d5e64f44053d4052bf1651dfb339497d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39465 } }
I20260927 14:52:07.103281 9122 leader_election.cc:302] T 00000000000000000000000000000000 P d5e64f44053d4052bf1651dfb339497d [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d5e64f44053d4052bf1651dfb339497d; no voters:
I20260927 14:52:07.103586 9122 leader_election.cc:288] T 00000000000000000000000000000000 P d5e64f44053d4052bf1651dfb339497d [CANDIDATE]: Term 2 election: Requested vote from peers
I20260927 14:52:07.103775 9126 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d5e64f44053d4052bf1651dfb339497d [term 2 FOLLOWER]: Leader election won for term 2
I20260927 14:52:07.104084 9126 raft_consensus.cc:686] T 00000000000000000000000000000000 P d5e64f44053d4052bf1651dfb339497d [term 2 LEADER]: Becoming Leader. State: Replica: d5e64f44053d4052bf1651dfb339497d, State: Running, Role: LEADER
I20260927 14:52:07.104511 9126 consensus_queue.cc:237] T 00000000000000000000000000000000 P d5e64f44053d4052bf1651dfb339497d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d5e64f44053d4052bf1651dfb339497d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39465 } }
I20260927 14:52:07.104756 9122 sys_catalog.cc:564] T 00000000000000000000000000000000 P d5e64f44053d4052bf1651dfb339497d [sys.catalog]: configured and running, proceeding with master startup.
I20260927 14:52:07.106058 9128 sys_catalog.cc:455] T 00000000000000000000000000000000 P d5e64f44053d4052bf1651dfb339497d [sys.catalog]: SysCatalogTable state changed. Reason: New leader d5e64f44053d4052bf1651dfb339497d. Latest consensus state: current_term: 2 leader_uuid: "d5e64f44053d4052bf1651dfb339497d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d5e64f44053d4052bf1651dfb339497d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39465 } } }
I20260927 14:52:07.106084 9127 sys_catalog.cc:455] T 00000000000000000000000000000000 P d5e64f44053d4052bf1651dfb339497d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "d5e64f44053d4052bf1651dfb339497d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d5e64f44053d4052bf1651dfb339497d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39465 } } }
I20260927 14:52:07.106175 9128 sys_catalog.cc:458] T 00000000000000000000000000000000 P d5e64f44053d4052bf1651dfb339497d [sys.catalog]: This master's current role is: LEADER
I20260927 14:52:07.106179 9127 sys_catalog.cc:458] T 00000000000000000000000000000000 P d5e64f44053d4052bf1651dfb339497d [sys.catalog]: This master's current role is: LEADER
I20260927 14:52:07.106496 9135 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 14:52:07.109711 9135 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 14:52:07.110425 9135 catalog_manager.cc:1259] Loaded cluster ID: f38914fb598340d58f9b35a5735e512f
I20260927 14:52:07.110524 9135 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 14:52:07.112782 9135 catalog_manager.cc:1504] Loading token signing keys...
I20260927 14:52:07.113497 9135 catalog_manager.cc:5879] T 00000000000000000000000000000000 P d5e64f44053d4052bf1651dfb339497d: Loaded TSK: 0
I20260927 14:52:07.114730 9135 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260927 14:52:07.188102 9124 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 14:52:07.188397 9124 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 14:52:07.188489 9124 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 14:52:07.192521 9124 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 14:52:07.192606 9124 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 14:52:07.192690 9124 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 14:52:07.197862 9124 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33137
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790520684602251-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790520684602251-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790520684602251-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:36023
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790520684602251-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=43299
--tserver_master_addrs=127.0.98.62:39465
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9124
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790520684602251-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:15:17 UTC on ccycloud.quasar-ledxfe.root.comops.site
build id 4676134
I20260927 14:52:07.198829 9124 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 14:52:07.199795 9124 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 14:52:07.209088 9149 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 14:52:07.209228 9124 server_base.cc:1034] running on GCE node
W20260927 14:52:07.209110 9148 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 14:52:07.209442 9151 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 14:52:07.209841 9124 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 14:52:07.210533 9124 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 14:52:07.211710 9124 hybrid_clock.cc:648] HybridClock initialized: now 1790520727211685 us; error 51 us; skew 500 ppm
Sep 27 14:52:07 dist-test-slave-xk1g krb5kdc[8541](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790520727, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 14:52:07.214473 9124 init.cc:377] Logged in from keytab as oryx/127.0.98.1@KRBTEST.COM (short username oryx)
I20260927 14:52:07.215400 9124 webserver.cc:460] Webserver started at http://127.0.98.1:43299/ using document root <none> and password file <none>
I20260927 14:52:07.215945 9124 fs_manager.cc:359] Metadata directory not provided
I20260927 14:52:07.216017 9124 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 14:52:07.219858 9124 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260927 14:52:07.221904 9158 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 14:52:07.223191 9124 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 14:52:07.223281 9124 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790520684602251-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790520684602251-392-0/minicluster-data/ts-0/wal
uuid: "fc33b763315b403784837872a2ce63a9"
format_stamp: "Formatted at 2026-09-27 14:52:06 on dist-test-slave-xk1g"
server_key: "cc729632985d97e3765d9a53a0aae2b5"
server_key_iv: "2a5986fa665f1635d591854fdfbe4409"
server_key_version: "encryptionkey@0"
I20260927 14:52:07.223837 9124 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790520684602251-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790520684602251-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790520684602251-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 14:52:07.239529 9124 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 14:52:07.242687 9124 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 14:52:07.242899 9124 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 14:52:07.243521 9124 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 14:52:07.244391 9124 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 14:52:07.244464 9124 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 14:52:07.244535 9124 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 14:52:07.244586 9124 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 14:52:07.253580 9124 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:36023
I20260927 14:52:07.253605 9271 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:36023 every 8 connection(s)
I20260927 14:52:07.254561 9124 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790520684602251-392-0/minicluster-data/ts-0/data/info.pb
I20260927 14:52:07.261302 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6936m5/build/debug/bin/kudu as pid 9124
I20260927 14:52:07.261744 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6936m5/build/debug/bin/kudu
/tmp/dist-test-task6936m5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790520684602251-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790520684602251-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790520684602251-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790520684602251-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:45779
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:39465
--webserver_port=37783
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790520684602251-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33137
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.98.2
--keytab_file=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790520684602251-392-0/krb5kdc/oryx_127.0.98.2.keytab with env {KRB5CCNAME=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790520684602251-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790520684602251-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790520684602251-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 14:52:07 dist-test-slave-xk1g krb5kdc[8541](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 14:52:07 dist-test-slave-xk1g krb5kdc[8541](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 14:52:07.268132 9275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:52:07.256767 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39465 (local address 127.0.98.1:51249)
0927 14:52:07.257308 (+ 541us) negotiation.cc:107] Waiting for socket to connect
0927 14:52:07.257349 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0927 14:52:07.258548 (+ 1199us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:52:07.259581 (+ 1033us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:52:07.259588 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:52:07.259929 (+ 341us) client_negotiation.cc:189] Negotiated authn=SASL
0927 14:52:07.260384 (+ 455us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:52:07.260398 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:52:07.261782 (+ 1384us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:52:07.261785 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:52:07.262829 (+ 1044us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:52:07.262843 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:52:07.264013 (+ 1170us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:52:07.264016 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:52:07.264155 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:52:07.264777 (+ 622us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 14:52:07.264794 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 14:52:07.267370 (+ 2576us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:39465: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":367,"thread_start_us":133,"threads_started":1}
W20260927 14:52:07.268708 9274 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 14:52:07.256605 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:51249 (local address 127.0.98.62:39465)
0927 14:52:07.257046 (+ 441us) server_negotiation.cc:206] Beginning negotiation
0927 14:52:07.257088 (+ 42us) server_negotiation.cc:399] Waiting for connection header
0927 14:52:07.257473 (+ 385us) server_negotiation.cc:407] Connection header received
0927 14:52:07.258920 (+ 1447us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:52:07.258931 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:52:07.259230 (+ 299us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:52:07.259495 (+ 265us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:52:07.260536 (+ 1041us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:52:07.261635 (+ 1099us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:52:07.263018 (+ 1383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:52:07.263846 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:52:07.264037 (+ 191us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:52:07.267692 (+ 3655us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:51249: BlockingRecv error: recv got EOF from 127.0.98.1:51249 (error 108)
Metrics: {"server-negotiator.queue_time_us":270,"thread_start_us":92,"threads_started":1}
W20260927 14:52:07.268749 9272 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:39465 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.98.62:39465: Client connection negotiation failed: client connection to 127.0.98.62:39465: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Sep 27 14:52:07 dist-test-slave-xk1g krb5kdc[8541](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 14:52:07 dist-test-slave-xk1g krb5kdc[8541](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 14:52:07.274766 9275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:52:07.269222 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39465 (local address 127.0.98.1:49557)
0927 14:52:07.269445 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0927 14:52:07.269461 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 14:52:07.269541 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:52:07.269825 (+ 284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:52:07.269827 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:52:07.269961 (+ 134us) client_negotiation.cc:189] Negotiated authn=SASL
0927 14:52:07.270046 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:52:07.270051 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:52:07.271146 (+ 1095us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:52:07.271148 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:52:07.271668 (+ 520us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:52:07.271675 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:52:07.272320 (+ 645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:52:07.272323 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:52:07.272389 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:52:07.272880 (+ 491us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 14:52:07.272891 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 14:52:07.274678 (+ 1787us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:39465: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":155}
W20260927 14:52:07.274953 9274 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 14:52:07.269433 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:49557 (local address 127.0.98.62:39465)
0927 14:52:07.269579 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0927 14:52:07.269585 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 14:52:07.269604 (+ 19us) server_negotiation.cc:407] Connection header received
0927 14:52:07.269648 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:52:07.269653 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:52:07.269714 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:52:07.269824 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:52:07.270154 (+ 330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:52:07.271039 (+ 885us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:52:07.271782 (+ 743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:52:07.272211 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:52:07.272336 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:52:07.274846 (+ 2510us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:49557: BlockingRecv error: recv got EOF from 127.0.98.1:49557 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
Sep 27 14:52:07 dist-test-slave-xk1g krb5kdc[8541](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 14:52:07 dist-test-slave-xk1g krb5kdc[8541](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 14:52:07.280323 9275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:52:07.275369 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39465 (local address 127.0.98.1:38283)
0927 14:52:07.275519 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0927 14:52:07.275533 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 14:52:07.275599 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:52:07.275821 (+ 222us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:52:07.275824 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:52:07.275980 (+ 156us) client_negotiation.cc:189] Negotiated authn=SASL
0927 14:52:07.276062 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:52:07.276068 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:52:07.276718 (+ 650us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:52:07.276721 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:52:07.277605 (+ 884us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:52:07.277615 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:52:07.278015 (+ 400us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:52:07.278018 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:52:07.278100 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:52:07.278537 (+ 437us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 14:52:07.278550 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0927 14:52:07.280240 (+ 1690us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:39465: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":92}
W20260927 14:52:07.280552 9272 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260927 14:52:07.280738 9274 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 14:52:07.275466 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:38283 (local address 127.0.98.62:39465)
0927 14:52:07.275580 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0927 14:52:07.275585 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 14:52:07.275603 (+ 18us) server_negotiation.cc:407] Connection header received
0927 14:52:07.275677 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:52:07.275679 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:52:07.275713 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:52:07.275775 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:52:07.276161 (+ 386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:52:07.276623 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:52:07.277708 (+ 1085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:52:07.277915 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:52:07.277988 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:52:07.280381 (+ 2393us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:38283: BlockingRecv error: recv got EOF from 127.0.98.1:38283 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
W20260927 14:52:07.374522 9276 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 14:52:07.374769 9276 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 14:52:07.374812 9276 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 14:52:07.378755 9276 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 14:52:07.378810 9276 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 14:52:07.378865 9276 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 14:52:07.383839 9276 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33137
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790520684602251-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790520684602251-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790520684602251-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:45779
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790520684602251-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=37783
--tserver_master_addrs=127.0.98.62:39465
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9276
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790520684602251-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:15:17 UTC on ccycloud.quasar-ledxfe.root.comops.site
build id 4676134
I20260927 14:52:07.384740 9276 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 14:52:07.385717 9276 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 14:52:07.394816 9282 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 14:52:07.394824 9281 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 14:52:07.394932 9276 server_base.cc:1034] running on GCE node
W20260927 14:52:07.394824 9284 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 14:52:07.395515 9276 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 14:52:07.396060 9276 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 14:52:07.397256 9276 hybrid_clock.cc:648] HybridClock initialized: now 1790520727397246 us; error 31 us; skew 500 ppm
Sep 27 14:52:07 dist-test-slave-xk1g krb5kdc[8541](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790520727, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 14:52:07.399847 9276 init.cc:377] Logged in from keytab as oryx/127.0.98.2@KRBTEST.COM (short username oryx)
I20260927 14:52:07.400736 9276 webserver.cc:460] Webserver started at http://127.0.98.2:37783/ using document root <none> and password file <none>
I20260927 14:52:07.401368 9276 fs_manager.cc:359] Metadata directory not provided
I20260927 14:52:07.401479 9276 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 14:52:07.405356 9276 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 14:52:07.407557 9291 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 14:52:07.408867 9276 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 14:52:07.409015 9276 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790520684602251-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790520684602251-392-0/minicluster-data/ts-1/wal
uuid: "afc62c555d7f44c0aee158e1d96ee9e7"
format_stamp: "Formatted at 2026-09-27 14:52:06 on dist-test-slave-xk1g"
server_key: "5c7feac7aa4d0a9750d418b176694780"
server_key_iv: "96401e5ab866647f5f12e8e9da901ccc"
server_key_version: "encryptionkey@0"
I20260927 14:52:07.409628 9276 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790520684602251-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790520684602251-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790520684602251-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 14:52:07.424675 9276 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 14:52:07.427654 9276 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 14:52:07.427851 9276 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 14:52:07.428447 9276 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 14:52:07.429322 9276 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 14:52:07.429390 9276 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 14:52:07.429461 9276 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 14:52:07.429503 9276 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 14:52:07.438381 9276 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:45779
I20260927 14:52:07.438410 9404 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:45779 every 8 connection(s)
I20260927 14:52:07.439373 9276 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790520684602251-392-0/minicluster-data/ts-1/data/info.pb
Sep 27 14:52:07 dist-test-slave-xk1g krb5kdc[8541](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 14:52:07.448501 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6936m5/build/debug/bin/kudu as pid 9276
I20260927 14:52:07.448904 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6936m5/build/debug/bin/kudu
/tmp/dist-test-task6936m5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790520684602251-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790520684602251-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790520684602251-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790520684602251-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:44407
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:39465
--webserver_port=33569
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790520684602251-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33137
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.98.3
--keytab_file=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790520684602251-392-0/krb5kdc/oryx_127.0.98.3.keytab with env {KRB5CCNAME=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790520684602251-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790520684602251-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790520684602251-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 14:52:07 dist-test-slave-xk1g krb5kdc[8541](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
W20260927 14:52:07.450104 9274 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 14:52:07.441163 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:59169 (local address 127.0.98.62:39465)
0927 14:52:07.441296 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0927 14:52:07.441300 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 14:52:07.442160 (+ 860us) server_negotiation.cc:407] Connection header received
0927 14:52:07.443497 (+ 1337us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:52:07.443501 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:52:07.443562 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:52:07.443669 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:52:07.444782 (+ 1113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:52:07.445355 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:52:07.446483 (+ 1128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:52:07.446699 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:52:07.446801 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:52:07.449987 (+ 3186us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:59169: BlockingRecv error: recv got EOF from 127.0.98.2:59169 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
I20260927 14:52:07.450776 9407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:52:07.441592 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39465 (local address 127.0.98.2:59169)
0927 14:52:07.442026 (+ 434us) negotiation.cc:107] Waiting for socket to connect
0927 14:52:07.442065 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0927 14:52:07.443276 (+ 1211us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:52:07.443836 (+ 560us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:52:07.443842 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:52:07.444185 (+ 343us) client_negotiation.cc:189] Negotiated authn=SASL
0927 14:52:07.444643 (+ 458us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:52:07.444657 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:52:07.445481 (+ 824us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:52:07.445485 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:52:07.446351 (+ 866us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:52:07.446360 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:52:07.446848 (+ 488us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:52:07.446852 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:52:07.447016 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:52:07.447593 (+ 577us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 14:52:07.447608 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 14:52:07.449878 (+ 2270us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:39465: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":256,"thread_start_us":110,"threads_started":1}
W20260927 14:52:07.451337 9405 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:39465 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.98.62:39465: Client connection negotiation failed: client connection to 127.0.98.62:39465: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Sep 27 14:52:07 dist-test-slave-xk1g krb5kdc[8541](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 14:52:07 dist-test-slave-xk1g krb5kdc[8541](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 14:52:07.457149 9407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:52:07.451759 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39465 (local address 127.0.98.2:53577)
0927 14:52:07.451868 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0927 14:52:07.451883 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 14:52:07.451962 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:52:07.452172 (+ 210us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:52:07.452175 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:52:07.452325 (+ 150us) client_negotiation.cc:189] Negotiated authn=SASL
0927 14:52:07.452407 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:52:07.452412 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:52:07.453212 (+ 800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:52:07.453217 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:52:07.454154 (+ 937us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:52:07.454163 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:52:07.454770 (+ 607us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:52:07.454774 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:52:07.454861 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:52:07.455357 (+ 496us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 14:52:07.455370 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0927 14:52:07.456994 (+ 1624us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:39465: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":45}
W20260927 14:52:07.457237 9274 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 14:52:07.451867 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:53577 (local address 127.0.98.62:39465)
0927 14:52:07.451972 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0927 14:52:07.451976 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 14:52:07.451990 (+ 14us) server_negotiation.cc:407] Connection header received
0927 14:52:07.452033 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:52:07.452035 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:52:07.452078 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:52:07.452159 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:52:07.452515 (+ 356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:52:07.453107 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:52:07.454261 (+ 1154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:52:07.454662 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:52:07.454770 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:52:07.457132 (+ 2362us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:53577: BlockingRecv error: recv got EOF from 127.0.98.2:53577 (error 108)
Metrics: {"server-negotiator.queue_time_us":34}
Sep 27 14:52:07 dist-test-slave-xk1g krb5kdc[8541](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 14:52:07 dist-test-slave-xk1g krb5kdc[8541](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 14:52:07.462253 9407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:52:07.457725 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39465 (local address 127.0.98.2:44669)
0927 14:52:07.457811 (+ 86us) negotiation.cc:107] Waiting for socket to connect
0927 14:52:07.457826 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 14:52:07.457891 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:52:07.458116 (+ 225us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:52:07.458119 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:52:07.458263 (+ 144us) client_negotiation.cc:189] Negotiated authn=SASL
0927 14:52:07.458322 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:52:07.458326 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:52:07.459000 (+ 674us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:52:07.459006 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:52:07.459506 (+ 500us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:52:07.459513 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:52:07.459937 (+ 424us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:52:07.459940 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:52:07.460016 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:52:07.460470 (+ 454us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 14:52:07.460481 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 14:52:07.462139 (+ 1658us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:39465: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":25}
W20260927 14:52:07.462443 9274 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 14:52:07.457826 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:44669 (local address 127.0.98.62:39465)
0927 14:52:07.457925 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0927 14:52:07.457931 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 14:52:07.457945 (+ 14us) server_negotiation.cc:407] Connection header received
0927 14:52:07.457979 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:52:07.457981 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:52:07.458014 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:52:07.458104 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:52:07.458432 (+ 328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:52:07.458901 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:52:07.459615 (+ 714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:52:07.459834 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:52:07.459935 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:52:07.462238 (+ 2303us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:44669: BlockingRecv error: recv got EOF from 127.0.98.2:44669 (error 108)
Metrics: {"server-negotiator.queue_time_us":32}
W20260927 14:52:07.462445 9405 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260927 14:52:07.559304 9408 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 14:52:07.559641 9408 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 14:52:07.559721 9408 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 14:52:07.564070 9408 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 14:52:07.564167 9408 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 14:52:07.564262 9408 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 14:52:07.569778 9408 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33137
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790520684602251-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790520684602251-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790520684602251-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:44407
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790520684602251-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=33569
--tserver_master_addrs=127.0.98.62:39465
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9408
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790520684602251-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:15:17 UTC on ccycloud.quasar-ledxfe.root.comops.site
build id 4676134
I20260927 14:52:07.570905 9408 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 14:52:07.571945 9408 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 14:52:07.581564 9414 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 14:52:07.581646 9408 server_base.cc:1034] running on GCE node
W20260927 14:52:07.581564 9413 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 14:52:07.581593 9416 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 14:52:07.582264 9408 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 14:52:07.582942 9408 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 14:52:07.584142 9408 hybrid_clock.cc:648] HybridClock initialized: now 1790520727584123 us; error 28 us; skew 500 ppm
Sep 27 14:52:07 dist-test-slave-xk1g krb5kdc[8541](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790520727, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 14:52:07.590787 9408 init.cc:377] Logged in from keytab as oryx/127.0.98.3@KRBTEST.COM (short username oryx)
I20260927 14:52:07.591816 9408 webserver.cc:460] Webserver started at http://127.0.98.3:33569/ using document root <none> and password file <none>
I20260927 14:52:07.592406 9408 fs_manager.cc:359] Metadata directory not provided
I20260927 14:52:07.592512 9408 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 14:52:07.596357 9408 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260927 14:52:07.598376 9423 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 14:52:07.599665 9408 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 14:52:07.599828 9408 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790520684602251-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790520684602251-392-0/minicluster-data/ts-2/wal
uuid: "d40da6e81dac4fb4ac5bc8ebd9208645"
format_stamp: "Formatted at 2026-09-27 14:52:06 on dist-test-slave-xk1g"
server_key: "c126f60b7b1533e2c787a916e715f1cd"
server_key_iv: "f95fdc71843c79d9f38f483149d41fbb"
server_key_version: "encryptionkey@0"
I20260927 14:52:07.600363 9408 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790520684602251-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790520684602251-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790520684602251-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 14:52:07.612739 9408 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 14:52:07.615698 9408 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 14:52:07.615896 9408 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 14:52:07.616446 9408 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 14:52:07.617332 9408 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 14:52:07.617399 9408 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 14:52:07.617468 9408 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 14:52:07.617520 9408 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 14:52:07.626652 9408 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:44407
I20260927 14:52:07.626673 9536 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:44407 every 8 connection(s)
I20260927 14:52:07.627707 9408 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790520684602251-392-0/minicluster-data/ts-2/data/info.pb
I20260927 14:52:07.635507 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6936m5/build/debug/bin/kudu as pid 9408
Sep 27 14:52:07 dist-test-slave-xk1g krb5kdc[8541](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 14:52:07 dist-test-slave-xk1g krb5kdc[8541](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
W20260927 14:52:07.638692 9274 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 14:52:07.629529 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:47521 (local address 127.0.98.62:39465)
0927 14:52:07.629657 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0927 14:52:07.629661 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 14:52:07.630433 (+ 772us) server_negotiation.cc:407] Connection header received
0927 14:52:07.631758 (+ 1325us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:52:07.631762 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:52:07.631821 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:52:07.631898 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:52:07.633040 (+ 1142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:52:07.633587 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:52:07.634708 (+ 1121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:52:07.634933 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:52:07.635042 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:52:07.638574 (+ 3532us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:47521: BlockingRecv error: recv got EOF from 127.0.98.3:47521 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
I20260927 14:52:07.639367 9539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:52:07.629880 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39465 (local address 127.0.98.3:47521)
0927 14:52:07.630282 (+ 402us) negotiation.cc:107] Waiting for socket to connect
0927 14:52:07.630321 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0927 14:52:07.631544 (+ 1223us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:52:07.632076 (+ 532us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:52:07.632082 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:52:07.632426 (+ 344us) client_negotiation.cc:189] Negotiated authn=SASL
0927 14:52:07.632891 (+ 465us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:52:07.632911 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:52:07.633711 (+ 800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:52:07.633717 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:52:07.634578 (+ 861us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:52:07.634587 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:52:07.635077 (+ 490us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:52:07.635081 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:52:07.635243 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:52:07.635831 (+ 588us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 14:52:07.635847 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 14:52:07.638471 (+ 2624us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:39465: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":230,"thread_start_us":96,"threads_started":1}
W20260927 14:52:07.639796 9537 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:39465 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.98.62:39465: Client connection negotiation failed: client connection to 127.0.98.62:39465: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Sep 27 14:52:07 dist-test-slave-xk1g krb5kdc[8541](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 14:52:07 dist-test-slave-xk1g krb5kdc[8541](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 14:52:07.645929 9539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:52:07.640241 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39465 (local address 127.0.98.3:33053)
0927 14:52:07.640437 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0927 14:52:07.640450 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 14:52:07.640556 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:52:07.640828 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:52:07.640831 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:52:07.641007 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0927 14:52:07.641172 (+ 165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:52:07.641181 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:52:07.642057 (+ 876us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:52:07.642061 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:52:07.642808 (+ 747us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:52:07.642817 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:52:07.643337 (+ 520us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:52:07.643340 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:52:07.643404 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:52:07.643827 (+ 423us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 14:52:07.643837 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0927 14:52:07.645686 (+ 1849us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:39465: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":140}
W20260927 14:52:07.646045 9274 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 14:52:07.640336 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:33053 (local address 127.0.98.62:39465)
0927 14:52:07.640432 (+ 96us) server_negotiation.cc:206] Beginning negotiation
0927 14:52:07.640436 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 14:52:07.640517 (+ 81us) server_negotiation.cc:407] Connection header received
0927 14:52:07.640679 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:52:07.640682 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:52:07.640718 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:52:07.640780 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:52:07.641295 (+ 515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:52:07.641947 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:52:07.642920 (+ 973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:52:07.643230 (+ 310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:52:07.643340 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:52:07.645916 (+ 2576us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:33053: BlockingRecv error: recv got EOF from 127.0.98.3:33053 (error 108)
Metrics: {"mutex_wait_us":34,"server-negotiator.queue_time_us":25}
I20260927 14:52:07.646060 9541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:52:07.636528 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50892 (local address 127.0.98.62:39465)
0927 14:52:07.636757 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0927 14:52:07.636761 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 14:52:07.636862 (+ 101us) server_negotiation.cc:407] Connection header received
0927 14:52:07.636969 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:52:07.636971 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:52:07.637012 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:52:07.637147 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:52:07.638085 (+ 938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:52:07.638621 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:52:07.639564 (+ 943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:52:07.639795 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:52:07.639956 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:52:07.641646 (+ 1690us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:52:07.641652 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:52:07.641673 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 14:52:07.641703 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:52:07.643674 (+ 1971us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:52:07.644123 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:52:07.644129 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:52:07.644135 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:52:07.644328 (+ 193us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:52:07.644621 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:52:07.644625 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:52:07.644626 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:52:07.645129 (+ 503us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:52:07.645363 (+ 234us) server_negotiation.cc:1035] Waiting for connection context
0927 14:52:07.645594 (+ 231us) server_negotiation.cc:299] Negotiation successful
0927 14:52:07.645933 (+ 339us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":56,"server-negotiator.queue_time_us":169,"thread_start_us":92,"threads_started":1}
Sep 27 14:52:07 dist-test-slave-xk1g krb5kdc[8541](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 14:52:07 dist-test-slave-xk1g krb5kdc[8541](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 14:52:07.651767 9542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:52:07.646461 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39465 (local address 127.0.98.3:56693)
0927 14:52:07.646794 (+ 333us) negotiation.cc:107] Waiting for socket to connect
0927 14:52:07.646809 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 14:52:07.646908 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:52:07.647125 (+ 217us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:52:07.647131 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:52:07.647296 (+ 165us) client_negotiation.cc:189] Negotiated authn=SASL
0927 14:52:07.647389 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:52:07.647394 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:52:07.648279 (+ 885us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:52:07.648282 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:52:07.649096 (+ 814us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:52:07.649106 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:52:07.649525 (+ 419us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:52:07.649528 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:52:07.649604 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:52:07.650032 (+ 428us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 14:52:07.650042 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0927 14:52:07.651647 (+ 1605us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:39465: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":253,"thread_start_us":118,"threads_started":1}
W20260927 14:52:07.651889 9541 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 14:52:07.646529 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:56693 (local address 127.0.98.62:39465)
0927 14:52:07.646674 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0927 14:52:07.646678 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 14:52:07.646888 (+ 210us) server_negotiation.cc:407] Connection header received
0927 14:52:07.646986 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:52:07.646988 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:52:07.647020 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:52:07.647109 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:52:07.647491 (+ 382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:52:07.648173 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:52:07.649212 (+ 1039us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:52:07.649427 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:52:07.649499 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:52:07.651760 (+ 2261us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:56693: BlockingRecv error: recv got EOF from 127.0.98.3:56693 (error 108)
Metrics: {"server-negotiator.queue_time_us":78}
W20260927 14:52:07.652068 9537 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Sep 27 14:52:08 dist-test-slave-xk1g krb5kdc[8541](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 14:52:08 dist-test-slave-xk1g krb5kdc[8541](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 14:52:08.286446 9543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:52:08.281212 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39465 (local address 127.0.98.1:50141)
0927 14:52:08.281501 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0927 14:52:08.281515 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 14:52:08.281601 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:52:08.281889 (+ 288us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:52:08.281892 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:52:08.282088 (+ 196us) client_negotiation.cc:189] Negotiated authn=SASL
0927 14:52:08.282196 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:52:08.282201 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:52:08.282950 (+ 749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:52:08.282952 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:52:08.283525 (+ 573us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:52:08.283532 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:52:08.283976 (+ 444us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:52:08.283979 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:52:08.284045 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:52:08.284522 (+ 477us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 14:52:08.284533 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 14:52:08.286323 (+ 1790us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:39465: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":201,"thread_start_us":109,"threads_started":1}
W20260927 14:52:08.286669 9544 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 14:52:08.281310 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:50141 (local address 127.0.98.62:39465)
0927 14:52:08.281557 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0927 14:52:08.281561 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 14:52:08.281574 (+ 13us) server_negotiation.cc:407] Connection header received
0927 14:52:08.281690 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:52:08.281692 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:52:08.281735 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:52:08.281825 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:52:08.282309 (+ 484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:52:08.282830 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:52:08.283638 (+ 808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:52:08.283879 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:52:08.283959 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:52:08.286516 (+ 2557us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:50141: BlockingRecv error: recv got EOF from 127.0.98.1:50141 (error 108)
Metrics: {"server-negotiator.queue_time_us":176,"thread_start_us":100,"threads_started":1}
Sep 27 14:52:08 dist-test-slave-xk1g krb5kdc[8541](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 14:52:08 dist-test-slave-xk1g krb5kdc[8541](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 14:52:08.468830 9545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:52:08.463116 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39465 (local address 127.0.98.2:55471)
0927 14:52:08.463403 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0927 14:52:08.463416 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 14:52:08.463522 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:52:08.463815 (+ 293us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:52:08.463817 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:52:08.464005 (+ 188us) client_negotiation.cc:189] Negotiated authn=SASL
0927 14:52:08.464116 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:52:08.464121 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:52:08.464905 (+ 784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:52:08.464909 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:52:08.465666 (+ 757us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:52:08.465676 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:52:08.466141 (+ 465us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:52:08.466143 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:52:08.466215 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:52:08.466697 (+ 482us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 14:52:08.466708 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 14:52:08.468656 (+ 1948us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:39465: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":205,"thread_start_us":92,"threads_started":1}
W20260927 14:52:08.469003 9544 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 14:52:08.463219 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:55471 (local address 127.0.98.62:39465)
0927 14:52:08.463433 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0927 14:52:08.463439 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 14:52:08.463475 (+ 36us) server_negotiation.cc:407] Connection header received
0927 14:52:08.463643 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:52:08.463646 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:52:08.463701 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:52:08.463788 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:52:08.464257 (+ 469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:52:08.464774 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:52:08.465821 (+ 1047us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:52:08.466039 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:52:08.466127 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:52:08.468851 (+ 2724us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:55471: BlockingRecv error: recv got EOF from 127.0.98.2:55471 (error 108)
Metrics: {"server-negotiator.queue_time_us":105}
Sep 27 14:52:08 dist-test-slave-xk1g krb5kdc[8541](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 14:52:08 dist-test-slave-xk1g krb5kdc[8541](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 14:52:08.659260 9546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:52:08.652638 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39465 (local address 127.0.98.3:43769)
0927 14:52:08.652944 (+ 306us) negotiation.cc:107] Waiting for socket to connect
0927 14:52:08.652957 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 14:52:08.653095 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:52:08.653416 (+ 321us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:52:08.653418 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:52:08.653578 (+ 160us) client_negotiation.cc:189] Negotiated authn=SASL
0927 14:52:08.653666 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:52:08.653671 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:52:08.654851 (+ 1180us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:52:08.654853 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:52:08.655409 (+ 556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:52:08.655415 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:52:08.656117 (+ 702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:52:08.656121 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:52:08.656212 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:52:08.656871 (+ 659us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 14:52:08.656891 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0927 14:52:08.659140 (+ 2249us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:39465: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":233,"thread_start_us":74,"threads_started":1}
W20260927 14:52:08.659502 9544 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 14:52:08.652761 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:43769 (local address 127.0.98.62:39465)
0927 14:52:08.652922 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0927 14:52:08.652929 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 14:52:08.653037 (+ 108us) server_negotiation.cc:407] Connection header received
0927 14:52:08.653194 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:52:08.653199 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:52:08.653265 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:52:08.653384 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:52:08.653780 (+ 396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:52:08.654740 (+ 960us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:52:08.655513 (+ 773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:52:08.655984 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:52:08.656104 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:52:08.659338 (+ 3234us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:43769: BlockingRecv error: recv got EOF from 127.0.98.3:43769 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
Sep 27 14:52:09 dist-test-slave-xk1g krb5kdc[8541](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 14:52:09 dist-test-slave-xk1g krb5kdc[8541](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 14:52:09.293129 9547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:52:09.287339 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39465 (local address 127.0.98.1:42085)
0927 14:52:09.287598 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0927 14:52:09.287611 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 14:52:09.287697 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:52:09.287981 (+ 284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:52:09.287983 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:52:09.288168 (+ 185us) client_negotiation.cc:189] Negotiated authn=SASL
0927 14:52:09.288285 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:52:09.288291 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:52:09.289100 (+ 809us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:52:09.289108 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:52:09.289717 (+ 609us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:52:09.289728 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:52:09.290337 (+ 609us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:52:09.290341 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:52:09.290441 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:52:09.291073 (+ 632us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 14:52:09.291086 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0927 14:52:09.292902 (+ 1816us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:39465: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":173,"thread_start_us":90,"threads_started":1}
W20260927 14:52:09.293218 9548 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 14:52:09.287403 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:42085 (local address 127.0.98.62:39465)
0927 14:52:09.287653 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0927 14:52:09.287656 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 14:52:09.287670 (+ 14us) server_negotiation.cc:407] Connection header received
0927 14:52:09.287781 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:52:09.287784 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:52:09.287824 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:52:09.287902 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:52:09.288394 (+ 492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:52:09.288930 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:52:09.289896 (+ 966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:52:09.290191 (+ 295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:52:09.290323 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:52:09.293079 (+ 2756us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:42085: BlockingRecv error: recv got EOF from 127.0.98.1:42085 (error 108)
Metrics: {"server-negotiator.queue_time_us":180,"thread_start_us":109,"threads_started":1}
Sep 27 14:52:09 dist-test-slave-xk1g krb5kdc[8541](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 14:52:09 dist-test-slave-xk1g krb5kdc[8541](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 14:52:09.475354 9549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:52:09.469681 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39465 (local address 127.0.98.2:59769)
0927 14:52:09.470002 (+ 321us) negotiation.cc:107] Waiting for socket to connect
0927 14:52:09.470019 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 14:52:09.470136 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:52:09.470407 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:52:09.470409 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:52:09.470610 (+ 201us) client_negotiation.cc:189] Negotiated authn=SASL
0927 14:52:09.470695 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:52:09.470701 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:52:09.471445 (+ 744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:52:09.471450 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:52:09.472003 (+ 553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:52:09.472010 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:52:09.472452 (+ 442us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:52:09.472456 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:52:09.472539 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:52:09.473101 (+ 562us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 14:52:09.473113 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 14:52:09.475225 (+ 2112us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:39465: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":215,"thread_start_us":70,"threads_started":1}
W20260927 14:52:09.475450 9548 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 14:52:09.469911 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:59769 (local address 127.0.98.62:39465)
0927 14:52:09.470048 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0927 14:52:09.470055 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 14:52:09.470071 (+ 16us) server_negotiation.cc:407] Connection header received
0927 14:52:09.470241 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:52:09.470245 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:52:09.470298 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:52:09.470383 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:52:09.470809 (+ 426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:52:09.471320 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:52:09.472118 (+ 798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:52:09.472339 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:52:09.472452 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:52:09.475342 (+ 2890us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:59769: BlockingRecv error: recv got EOF from 127.0.98.2:59769 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Sep 27 14:52:09 dist-test-slave-xk1g krb5kdc[8541](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 14:52:09 dist-test-slave-xk1g krb5kdc[8541](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 14:52:09.665151 9550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:52:09.660040 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39465 (local address 127.0.98.3:42767)
0927 14:52:09.660302 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0927 14:52:09.660318 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 14:52:09.660423 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:52:09.660677 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:52:09.660679 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:52:09.660871 (+ 192us) client_negotiation.cc:189] Negotiated authn=SASL
0927 14:52:09.660997 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:52:09.661002 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:52:09.661741 (+ 739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:52:09.661743 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:52:09.662304 (+ 561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:52:09.662312 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:52:09.662705 (+ 393us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:52:09.662708 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:52:09.662777 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:52:09.663257 (+ 480us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 14:52:09.663268 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 14:52:09.664972 (+ 1704us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:39465: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":101,"threads_started":1}
W20260927 14:52:09.665211 9548 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 14:52:09.660179 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:42767 (local address 127.0.98.62:39465)
0927 14:52:09.660302 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0927 14:52:09.660305 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 14:52:09.660403 (+ 98us) server_negotiation.cc:407] Connection header received
0927 14:52:09.660490 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:52:09.660492 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:52:09.660554 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:52:09.660634 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:52:09.661117 (+ 483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:52:09.661653 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:52:09.662408 (+ 755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:52:09.662613 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:52:09.662686 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:52:09.665125 (+ 2439us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:42767: BlockingRecv error: recv got EOF from 127.0.98.3:42767 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
Sep 27 14:52:10 dist-test-slave-xk1g krb5kdc[8541](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 14:52:10 dist-test-slave-xk1g krb5kdc[8541](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 14:52:10.299016 9551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:52:10.293940 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39465 (local address 127.0.98.1:60011)
0927 14:52:10.294228 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0927 14:52:10.294243 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 14:52:10.294332 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:52:10.294571 (+ 239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:52:10.294573 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:52:10.294751 (+ 178us) client_negotiation.cc:189] Negotiated authn=SASL
0927 14:52:10.294842 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:52:10.294847 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:52:10.295574 (+ 727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:52:10.295578 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:52:10.296157 (+ 579us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:52:10.296166 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:52:10.296561 (+ 395us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:52:10.296563 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:52:10.296623 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:52:10.297153 (+ 530us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 14:52:10.297171 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0927 14:52:10.298908 (+ 1737us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:39465: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":202,"thread_start_us":84,"threads_started":1}
W20260927 14:52:10.299171 9552 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 14:52:10.294061 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:60011 (local address 127.0.98.62:39465)
0927 14:52:10.294269 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0927 14:52:10.294273 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 14:52:10.294286 (+ 13us) server_negotiation.cc:407] Connection header received
0927 14:52:10.294419 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:52:10.294421 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:52:10.294462 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:52:10.294535 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:52:10.294940 (+ 405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:52:10.295452 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:52:10.296258 (+ 806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:52:10.296468 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:52:10.296540 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:52:10.299036 (+ 2496us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:60011: BlockingRecv error: recv got EOF from 127.0.98.1:60011 (error 108)
Metrics: {"server-negotiator.queue_time_us":149,"thread_start_us":54,"threads_started":1}
Sep 27 14:52:10 dist-test-slave-xk1g krb5kdc[8541](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 14:52:10 dist-test-slave-xk1g krb5kdc[8541](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 14:52:10.482115 9553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:52:10.476349 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39465 (local address 127.0.98.2:52375)
0927 14:52:10.476648 (+ 299us) negotiation.cc:107] Waiting for socket to connect
0927 14:52:10.476667 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 14:52:10.476778 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:52:10.477111 (+ 333us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:52:10.477114 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:52:10.477306 (+ 192us) client_negotiation.cc:189] Negotiated authn=SASL
0927 14:52:10.477439 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:52:10.477450 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:52:10.478363 (+ 913us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:52:10.478365 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:52:10.478918 (+ 553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:52:10.478925 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:52:10.479335 (+ 410us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:52:10.479337 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:52:10.479394 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:52:10.479953 (+ 559us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 14:52:10.479963 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0927 14:52:10.481986 (+ 2023us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:39465: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":207,"thread_start_us":112,"threads_started":1}
W20260927 14:52:10.482358 9552 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 14:52:10.476418 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:52375 (local address 127.0.98.62:39465)
0927 14:52:10.476581 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0927 14:52:10.476588 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 14:52:10.476759 (+ 171us) server_negotiation.cc:407] Connection header received
0927 14:52:10.476888 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:52:10.476890 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:52:10.476933 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:52:10.477037 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:52:10.477578 (+ 541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:52:10.478261 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:52:10.479024 (+ 763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:52:10.479238 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:52:10.479323 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:52:10.482198 (+ 2875us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:52375: BlockingRecv error: recv got EOF from 127.0.98.2:52375 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
Sep 27 14:52:10 dist-test-slave-xk1g krb5kdc[8541](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 14:52:10 dist-test-slave-xk1g krb5kdc[8541](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 14:52:10.671496 9554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:52:10.666038 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39465 (local address 127.0.98.3:45123)
0927 14:52:10.666374 (+ 336us) negotiation.cc:107] Waiting for socket to connect
0927 14:52:10.666387 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 14:52:10.666521 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:52:10.666787 (+ 266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:52:10.666789 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:52:10.666994 (+ 205us) client_negotiation.cc:189] Negotiated authn=SASL
0927 14:52:10.667081 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:52:10.667086 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:52:10.667837 (+ 751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:52:10.667843 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:52:10.668410 (+ 567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:52:10.668417 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:52:10.668871 (+ 454us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:52:10.668874 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:52:10.668956 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:52:10.669516 (+ 560us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 14:52:10.669528 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 14:52:10.671384 (+ 1856us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:39465: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":243,"thread_start_us":86,"threads_started":1}
W20260927 14:52:10.671659 9552 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 14:52:10.666158 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:45123 (local address 127.0.98.62:39465)
0927 14:52:10.666307 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0927 14:52:10.666312 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 14:52:10.666463 (+ 151us) server_negotiation.cc:407] Connection header received
0927 14:52:10.666620 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:52:10.666624 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:52:10.666678 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:52:10.666763 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:52:10.667199 (+ 436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:52:10.667711 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:52:10.668531 (+ 820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:52:10.668764 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:52:10.668871 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:52:10.671512 (+ 2641us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:45123: BlockingRecv error: recv got EOF from 127.0.98.3:45123 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
Sep 27 14:52:11 dist-test-slave-xk1g krb5kdc[8541](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 14:52:11 dist-test-slave-xk1g krb5kdc[8541](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 14:52:11.311465 9555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:52:11.299983 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39465 (local address 127.0.98.1:54169)
0927 14:52:11.300319 (+ 336us) negotiation.cc:107] Waiting for socket to connect
0927 14:52:11.300334 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 14:52:11.300461 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:52:11.300712 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:52:11.300714 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:52:11.300896 (+ 182us) client_negotiation.cc:189] Negotiated authn=SASL
0927 14:52:11.301003 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:52:11.301008 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:52:11.301882 (+ 874us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:52:11.301885 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:52:11.302438 (+ 553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:52:11.302445 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:52:11.302866 (+ 421us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:52:11.302869 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:52:11.302937 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:52:11.309249 (+ 6312us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 14:52:11.309262 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0927 14:52:11.311276 (+ 2014us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:39465: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":259,"thread_start_us":92,"threads_started":1}
W20260927 14:52:11.311532 9556 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 14:52:11.300120 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:54169 (local address 127.0.98.62:39465)
0927 14:52:11.300352 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0927 14:52:11.300355 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 14:52:11.300434 (+ 79us) server_negotiation.cc:407] Connection header received
0927 14:52:11.300548 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:52:11.300550 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:52:11.300591 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:52:11.300666 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:52:11.301197 (+ 531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:52:11.301742 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:52:11.302537 (+ 795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:52:11.302746 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:52:11.302818 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:52:11.311411 (+ 8593us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:54169: BlockingRecv error: recv got EOF from 127.0.98.1:54169 (error 108)
Metrics: {"server-negotiator.queue_time_us":148,"thread_start_us":55,"threads_started":1}
Sep 27 14:52:11 dist-test-slave-xk1g krb5kdc[8541](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 14:52:11 dist-test-slave-xk1g krb5kdc[8541](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 14:52:11.487782 9557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:52:11.482812 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39465 (local address 127.0.98.2:54949)
0927 14:52:11.483094 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0927 14:52:11.483108 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 14:52:11.483205 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:52:11.483441 (+ 236us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:52:11.483443 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:52:11.483617 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0927 14:52:11.483714 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:52:11.483720 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:52:11.484400 (+ 680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:52:11.484402 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:52:11.484937 (+ 535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:52:11.484945 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:52:11.485371 (+ 426us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:52:11.485374 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:52:11.485453 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:52:11.485943 (+ 490us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 14:52:11.485954 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 14:52:11.487656 (+ 1702us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:39465: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":212,"thread_start_us":94,"threads_started":1}
W20260927 14:52:11.487910 9556 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 14:52:11.482911 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:54949 (local address 127.0.98.62:39465)
0927 14:52:11.483081 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0927 14:52:11.483086 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 14:52:11.483172 (+ 86us) server_negotiation.cc:407] Connection header received
0927 14:52:11.483279 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:52:11.483281 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:52:11.483323 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:52:11.483396 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:52:11.483807 (+ 411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:52:11.484312 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:52:11.485031 (+ 719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:52:11.485265 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:52:11.485340 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:52:11.487798 (+ 2458us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:54949: BlockingRecv error: recv got EOF from 127.0.98.2:54949 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
Sep 27 14:52:11 dist-test-slave-xk1g krb5kdc[8541](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 14:52:11 dist-test-slave-xk1g krb5kdc[8541](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 14:52:11.677379 9558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:52:11.672247 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39465 (local address 127.0.98.3:39409)
0927 14:52:11.672530 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0927 14:52:11.672545 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 14:52:11.672652 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:52:11.672892 (+ 240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:52:11.672894 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:52:11.673109 (+ 215us) client_negotiation.cc:189] Negotiated authn=SASL
0927 14:52:11.673221 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:52:11.673228 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:52:11.673918 (+ 690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:52:11.673920 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:52:11.674496 (+ 576us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:52:11.674504 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:52:11.674889 (+ 385us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:52:11.674892 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:52:11.674956 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:52:11.675448 (+ 492us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 14:52:11.675461 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0927 14:52:11.677234 (+ 1773us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:39465: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":102,"threads_started":1}
W20260927 14:52:11.677465 9556 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 14:52:11.672321 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:39409 (local address 127.0.98.62:39465)
0927 14:52:11.672448 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0927 14:52:11.672458 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 14:52:11.672618 (+ 160us) server_negotiation.cc:407] Connection header received
0927 14:52:11.672739 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:52:11.672741 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:52:11.672783 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:52:11.672849 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:52:11.673325 (+ 476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:52:11.673819 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:52:11.674598 (+ 779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:52:11.674797 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:52:11.674867 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:52:11.677351 (+ 2484us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:39409: BlockingRecv error: recv got EOF from 127.0.98.3:39409 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
Sep 27 14:52:12 dist-test-slave-xk1g krb5kdc[8541](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 14:52:12 dist-test-slave-xk1g krb5kdc[8541](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 14:52:12.317996 9559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:52:12.312331 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39465 (local address 127.0.98.1:59287)
0927 14:52:12.312604 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0927 14:52:12.312617 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 14:52:12.312739 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:52:12.313022 (+ 283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:52:12.313025 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:52:12.313224 (+ 199us) client_negotiation.cc:189] Negotiated authn=SASL
0927 14:52:12.313321 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:52:12.313326 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:52:12.314165 (+ 839us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:52:12.314184 (+ 19us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:52:12.314766 (+ 582us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:52:12.314776 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:52:12.315280 (+ 504us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:52:12.315284 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:52:12.315381 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:52:12.315900 (+ 519us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 14:52:12.315914 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0927 14:52:12.317811 (+ 1897us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:39465: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":108,"threads_started":1}
W20260927 14:52:12.318077 9560 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 14:52:12.312392 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:59287 (local address 127.0.98.62:39465)
0927 14:52:12.312656 (+ 264us) server_negotiation.cc:206] Beginning negotiation
0927 14:52:12.312659 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 14:52:12.312711 (+ 52us) server_negotiation.cc:407] Connection header received
0927 14:52:12.312836 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:52:12.312838 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:52:12.312886 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:52:12.312991 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:52:12.313419 (+ 428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:52:12.314040 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:52:12.314905 (+ 865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:52:12.315144 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:52:12.315265 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:52:12.317946 (+ 2681us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:59287: BlockingRecv error: recv got EOF from 127.0.98.1:59287 (error 108)
Metrics: {"server-negotiator.queue_time_us":197,"thread_start_us":117,"threads_started":1}
Sep 27 14:52:12 dist-test-slave-xk1g krb5kdc[8541](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 14:52:12 dist-test-slave-xk1g krb5kdc[8541](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 14:52:12.494660 9561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:52:12.488728 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39465 (local address 127.0.98.2:33059)
0927 14:52:12.489031 (+ 303us) negotiation.cc:107] Waiting for socket to connect
0927 14:52:12.489045 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 14:52:12.489146 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:52:12.489355 (+ 209us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:52:12.489358 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:52:12.489534 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0927 14:52:12.489679 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:52:12.489690 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:52:12.490867 (+ 1177us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:52:12.490870 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:52:12.491499 (+ 629us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:52:12.491508 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:52:12.491928 (+ 420us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:52:12.491931 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:52:12.491990 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:52:12.492464 (+ 474us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 14:52:12.492475 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 14:52:12.494539 (+ 2064us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:39465: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":214,"thread_start_us":127,"threads_started":1}
W20260927 14:52:12.494773 9560 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 14:52:12.488942 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:33059 (local address 127.0.98.62:39465)
0927 14:52:12.489145 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0927 14:52:12.489149 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 14:52:12.489163 (+ 14us) server_negotiation.cc:407] Connection header received
0927 14:52:12.489203 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:52:12.489205 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:52:12.489247 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:52:12.489318 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:52:12.489849 (+ 531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:52:12.490732 (+ 883us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:52:12.491601 (+ 869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:52:12.491832 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:52:12.491910 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:52:12.494663 (+ 2753us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:33059: BlockingRecv error: recv got EOF from 127.0.98.2:33059 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
Sep 27 14:52:12 dist-test-slave-xk1g krb5kdc[8541](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 14:52:12 dist-test-slave-xk1g krb5kdc[8541](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 14:52:12.683344 9562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:52:12.678130 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39465 (local address 127.0.98.3:35819)
0927 14:52:12.678438 (+ 308us) negotiation.cc:107] Waiting for socket to connect
0927 14:52:12.678451 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 14:52:12.678582 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:52:12.678831 (+ 249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:52:12.678833 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:52:12.679003 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0927 14:52:12.679093 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:52:12.679097 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:52:12.679782 (+ 685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:52:12.679799 (+ 17us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:52:12.680363 (+ 564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:52:12.680371 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:52:12.680773 (+ 402us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:52:12.680776 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:52:12.680847 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:52:12.681417 (+ 570us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 14:52:12.681437 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0927 14:52:12.683224 (+ 1787us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:39465: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":236,"thread_start_us":98,"threads_started":1}
W20260927 14:52:12.683486 9560 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 14:52:12.678181 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:35819 (local address 127.0.98.62:39465)
0927 14:52:12.678323 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0927 14:52:12.678327 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 14:52:12.678564 (+ 237us) server_negotiation.cc:407] Connection header received
0927 14:52:12.678674 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:52:12.678676 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:52:12.678720 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:52:12.678784 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:52:12.679190 (+ 406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:52:12.679696 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:52:12.680465 (+ 769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:52:12.680684 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:52:12.680756 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:52:12.683355 (+ 2599us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:35819: BlockingRecv error: recv got EOF from 127.0.98.3:35819 (error 108)
Metrics: {"server-negotiator.queue_time_us":35}
Sep 27 14:52:13 dist-test-slave-xk1g krb5kdc[8541](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 14:52:13 dist-test-slave-xk1g krb5kdc[8541](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 14:52:13.324205 9563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:52:13.318825 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39465 (local address 127.0.98.1:50877)
0927 14:52:13.319107 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0927 14:52:13.319125 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 14:52:13.319238 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:52:13.319476 (+ 238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:52:13.319479 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:52:13.319655 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0927 14:52:13.319749 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:52:13.319754 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:52:13.320520 (+ 766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:52:13.320523 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:52:13.321102 (+ 579us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:52:13.321110 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:52:13.321584 (+ 474us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:52:13.321588 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:52:13.321673 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:52:13.322280 (+ 607us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 14:52:13.322299 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0927 14:52:13.324090 (+ 1791us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:39465: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":71,"threads_started":1}
W20260927 14:52:13.324496 9564 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 14:52:13.318975 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:50877 (local address 127.0.98.62:39465)
0927 14:52:13.319228 (+ 253us) server_negotiation.cc:206] Beginning negotiation
0927 14:52:13.319231 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 14:52:13.319247 (+ 16us) server_negotiation.cc:407] Connection header received
0927 14:52:13.319313 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:52:13.319315 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:52:13.319358 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:52:13.319436 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:52:13.319871 (+ 435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:52:13.320424 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:52:13.321226 (+ 802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:52:13.321464 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:52:13.321581 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:52:13.324222 (+ 2641us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:50877: BlockingRecv error: recv got EOF from 127.0.98.1:50877 (error 108)
Metrics: {"server-negotiator.queue_time_us":183,"thread_start_us":91,"threads_started":1}
Sep 27 14:52:13 dist-test-slave-xk1g krb5kdc[8541](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 14:52:13 dist-test-slave-xk1g krb5kdc[8541](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 14:52:13.500332 9565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:52:13.495406 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39465 (local address 127.0.98.2:54229)
0927 14:52:13.495660 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0927 14:52:13.495674 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 14:52:13.495757 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:52:13.495958 (+ 201us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:52:13.495960 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:52:13.496132 (+ 172us) client_negotiation.cc:189] Negotiated authn=SASL
0927 14:52:13.496225 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:52:13.496230 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:52:13.496919 (+ 689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:52:13.496921 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:52:13.497500 (+ 579us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:52:13.497508 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:52:13.497886 (+ 378us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:52:13.497889 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:52:13.497945 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:52:13.498506 (+ 561us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 14:52:13.498518 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 14:52:13.500211 (+ 1693us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:39465: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":172,"thread_start_us":92,"threads_started":1}
W20260927 14:52:13.500422 9564 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 14:52:13.495622 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:54229 (local address 127.0.98.62:39465)
0927 14:52:13.495764 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0927 14:52:13.495768 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 14:52:13.495780 (+ 12us) server_negotiation.cc:407] Connection header received
0927 14:52:13.495814 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:52:13.495816 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:52:13.495855 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:52:13.495924 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:52:13.496321 (+ 397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:52:13.496831 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:52:13.497592 (+ 761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:52:13.497795 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:52:13.497868 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:52:13.500329 (+ 2461us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:54229: BlockingRecv error: recv got EOF from 127.0.98.2:54229 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
Sep 27 14:52:13 dist-test-slave-xk1g krb5kdc[8541](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 14:52:13 dist-test-slave-xk1g krb5kdc[8541](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 14:52:13.689520 9566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:52:13.684036 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39465 (local address 127.0.98.3:46143)
0927 14:52:13.684273 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0927 14:52:13.684307 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0927 14:52:13.684402 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:52:13.684612 (+ 210us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:52:13.684614 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:52:13.684783 (+ 169us) client_negotiation.cc:189] Negotiated authn=SASL
0927 14:52:13.684932 (+ 149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:52:13.684940 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:52:13.685984 (+ 1044us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:52:13.685988 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:52:13.686635 (+ 647us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:52:13.686645 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:52:13.687049 (+ 404us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:52:13.687052 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:52:13.687129 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:52:13.687671 (+ 542us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 14:52:13.687694 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0927 14:52:13.689409 (+ 1715us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:39465: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":161,"thread_start_us":87,"threads_started":1}
W20260927 14:52:13.689672 9564 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 14:52:13.684142 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:46143 (local address 127.0.98.62:39465)
0927 14:52:13.684265 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0927 14:52:13.684268 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 14:52:13.684374 (+ 106us) server_negotiation.cc:407] Connection header received
0927 14:52:13.684468 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:52:13.684470 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:52:13.684512 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:52:13.684580 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:52:13.685093 (+ 513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:52:13.685862 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:52:13.686742 (+ 880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:52:13.686957 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:52:13.687032 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:52:13.689528 (+ 2496us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:46143: BlockingRecv error: recv got EOF from 127.0.98.3:46143 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
Sep 27 14:52:14 dist-test-slave-xk1g krb5kdc[8541](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 14:52:14 dist-test-slave-xk1g krb5kdc[8541](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 14:52:14.330536 9567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:52:14.324963 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39465 (local address 127.0.98.1:56539)
0927 14:52:14.325322 (+ 359us) negotiation.cc:107] Waiting for socket to connect
0927 14:52:14.325335 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 14:52:14.325429 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:52:14.325641 (+ 212us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:52:14.325643 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:52:14.325815 (+ 172us) client_negotiation.cc:189] Negotiated authn=SASL
0927 14:52:14.325918 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:52:14.325923 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:52:14.326728 (+ 805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:52:14.326731 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:52:14.327395 (+ 664us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:52:14.327404 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:52:14.327882 (+ 478us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:52:14.327885 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:52:14.328011 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:52:14.328624 (+ 613us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 14:52:14.328636 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 14:52:14.330426 (+ 1790us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:39465: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":269,"thread_start_us":157,"threads_started":1}
W20260927 14:52:14.330662 9568 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 14:52:14.325047 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:56539 (local address 127.0.98.62:39465)
0927 14:52:14.325318 (+ 271us) server_negotiation.cc:206] Beginning negotiation
0927 14:52:14.325321 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 14:52:14.325400 (+ 79us) server_negotiation.cc:407] Connection header received
0927 14:52:14.325492 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:52:14.325494 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:52:14.325536 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:52:14.325609 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:52:14.326008 (+ 399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:52:14.326616 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:52:14.327542 (+ 926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:52:14.327761 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:52:14.327837 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:52:14.330539 (+ 2702us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:56539: BlockingRecv error: recv got EOF from 127.0.98.1:56539 (error 108)
Metrics: {"server-negotiator.queue_time_us":171,"thread_start_us":74,"threads_started":1}
Sep 27 14:52:14 dist-test-slave-xk1g krb5kdc[8541](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 14:52:14 dist-test-slave-xk1g krb5kdc[8541](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 14:52:14.506923 9569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:52:14.501271 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39465 (local address 127.0.98.2:35543)
0927 14:52:14.501573 (+ 302us) negotiation.cc:107] Waiting for socket to connect
0927 14:52:14.501584 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 14:52:14.501714 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:52:14.501987 (+ 273us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:52:14.501990 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:52:14.502190 (+ 200us) client_negotiation.cc:189] Negotiated authn=SASL
0927 14:52:14.502278 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:52:14.502283 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:52:14.503024 (+ 741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:52:14.503030 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:52:14.503653 (+ 623us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:52:14.503664 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:52:14.504118 (+ 454us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:52:14.504122 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:52:14.504201 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:52:14.504751 (+ 550us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 14:52:14.504766 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 14:52:14.506754 (+ 1988us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:39465: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":226,"thread_start_us":80,"threads_started":1}
W20260927 14:52:14.507020 9568 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 14:52:14.501327 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:35543 (local address 127.0.98.62:39465)
0927 14:52:14.501461 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0927 14:52:14.501466 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 14:52:14.501655 (+ 189us) server_negotiation.cc:407] Connection header received
0927 14:52:14.501815 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:52:14.501818 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:52:14.501881 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:52:14.501967 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:52:14.502386 (+ 419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:52:14.502918 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:52:14.503776 (+ 858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:52:14.504004 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:52:14.504115 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:52:14.506911 (+ 2796us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:35543: BlockingRecv error: recv got EOF from 127.0.98.2:35543 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
Sep 27 14:52:14 dist-test-slave-xk1g krb5kdc[8541](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 14:52:14 dist-test-slave-xk1g krb5kdc[8541](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 14:52:14.695724 9570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:52:14.690286 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39465 (local address 127.0.98.3:59553)
0927 14:52:14.690648 (+ 362us) negotiation.cc:107] Waiting for socket to connect
0927 14:52:14.690664 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 14:52:14.690776 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:52:14.691054 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:52:14.691057 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:52:14.691266 (+ 209us) client_negotiation.cc:189] Negotiated authn=SASL
0927 14:52:14.691357 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:52:14.691363 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:52:14.692091 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:52:14.692097 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:52:14.692668 (+ 571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:52:14.692676 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:52:14.693284 (+ 608us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:52:14.693288 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:52:14.693364 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:52:14.693850 (+ 486us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 14:52:14.693861 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 14:52:14.695559 (+ 1698us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:39465: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":287,"thread_start_us":124,"threads_started":1}
W20260927 14:52:14.695789 9568 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 14:52:14.690472 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:59553 (local address 127.0.98.62:39465)
0927 14:52:14.690624 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0927 14:52:14.690629 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 14:52:14.690722 (+ 93us) server_negotiation.cc:407] Connection header received
0927 14:52:14.690884 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:52:14.690890 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:52:14.690946 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:52:14.691030 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:52:14.691473 (+ 443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:52:14.691988 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:52:14.692787 (+ 799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:52:14.693015 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:52:14.693147 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:52:14.695683 (+ 2536us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:59553: BlockingRecv error: recv got EOF from 127.0.98.3:59553 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
Sep 27 14:52:15 dist-test-slave-xk1g krb5kdc[8541](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 14:52:15 dist-test-slave-xk1g krb5kdc[8541](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 14:52:15.336802 9571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:52:15.331319 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39465 (local address 127.0.98.1:34933)
0927 14:52:15.331684 (+ 365us) negotiation.cc:107] Waiting for socket to connect
0927 14:52:15.331697 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 14:52:15.331790 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:52:15.332065 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:52:15.332067 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:52:15.332233 (+ 166us) client_negotiation.cc:189] Negotiated authn=SASL
0927 14:52:15.332321 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:52:15.332326 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:52:15.333205 (+ 879us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:52:15.333208 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:52:15.333759 (+ 551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:52:15.333766 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:52:15.334198 (+ 432us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:52:15.334200 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:52:15.334258 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:52:15.334767 (+ 509us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 14:52:15.334778 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 14:52:15.336661 (+ 1883us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:39465: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":234,"thread_start_us":112,"threads_started":1}
W20260927 14:52:15.336849 9572 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 14:52:15.331449 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:34933 (local address 127.0.98.62:39465)
0927 14:52:15.331701 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0927 14:52:15.331704 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 14:52:15.331745 (+ 41us) server_negotiation.cc:407] Connection header received
0927 14:52:15.331876 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:52:15.331878 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:52:15.331924 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:52:15.332034 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:52:15.332439 (+ 405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:52:15.332998 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:52:15.333874 (+ 876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:52:15.334100 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:52:15.334184 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:52:15.336751 (+ 2567us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:34933: BlockingRecv error: recv got EOF from 127.0.98.1:34933 (error 108)
Metrics: {"server-negotiator.queue_time_us":155,"thread_start_us":60,"threads_started":1}
Sep 27 14:52:15 dist-test-slave-xk1g krb5kdc[8541](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 14:52:15 dist-test-slave-xk1g krb5kdc[8541](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 14:52:15.513387 9573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:52:15.507760 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39465 (local address 127.0.98.2:58701)
0927 14:52:15.508046 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0927 14:52:15.508058 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 14:52:15.508191 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:52:15.508465 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:52:15.508468 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:52:15.508670 (+ 202us) client_negotiation.cc:189] Negotiated authn=SASL
0927 14:52:15.508763 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:52:15.508777 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:52:15.509757 (+ 980us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:52:15.509762 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:52:15.510331 (+ 569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:52:15.510340 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:52:15.510802 (+ 462us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:52:15.510805 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:52:15.510907 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:52:15.511433 (+ 526us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 14:52:15.511445 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 14:52:15.513223 (+ 1778us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:39465: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":210,"thread_start_us":75,"threads_started":1}
W20260927 14:52:15.513444 9572 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 14:52:15.507823 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:58701 (local address 127.0.98.62:39465)
0927 14:52:15.507975 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0927 14:52:15.507980 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 14:52:15.508135 (+ 155us) server_negotiation.cc:407] Connection header received
0927 14:52:15.508299 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:52:15.508308 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:52:15.508352 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:52:15.508438 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:52:15.508896 (+ 458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:52:15.509644 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:52:15.510462 (+ 818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:52:15.510681 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:52:15.510789 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:52:15.513340 (+ 2551us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:58701: BlockingRecv error: recv got EOF from 127.0.98.2:58701 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
Sep 27 14:52:15 dist-test-slave-xk1g krb5kdc[8541](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 14:52:15 dist-test-slave-xk1g krb5kdc[8541](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 14:52:15.702179 9574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:52:15.696513 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39465 (local address 127.0.98.3:50341)
0927 14:52:15.696817 (+ 304us) negotiation.cc:107] Waiting for socket to connect
0927 14:52:15.696835 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 14:52:15.696973 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:52:15.697269 (+ 296us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:52:15.697272 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:52:15.697499 (+ 227us) client_negotiation.cc:189] Negotiated authn=SASL
0927 14:52:15.697588 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:52:15.697593 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:52:15.698486 (+ 893us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:52:15.698492 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:52:15.699064 (+ 572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:52:15.699071 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:52:15.699532 (+ 461us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:52:15.699536 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:52:15.699621 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:52:15.700138 (+ 517us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 14:52:15.700149 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 14:52:15.702010 (+ 1861us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:39465: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":225,"thread_start_us":107,"threads_started":1}
W20260927 14:52:15.702239 9572 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 14:52:15.696584 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:50341 (local address 127.0.98.62:39465)
0927 14:52:15.696758 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0927 14:52:15.696764 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 14:52:15.696923 (+ 159us) server_negotiation.cc:407] Connection header received
0927 14:52:15.697104 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:52:15.697107 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:52:15.697157 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:52:15.697249 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:52:15.697706 (+ 457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:52:15.698380 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:52:15.699180 (+ 800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:52:15.699421 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:52:15.699531 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:52:15.702134 (+ 2603us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:50341: BlockingRecv error: recv got EOF from 127.0.98.3:50341 (error 108)
Metrics: {"server-negotiator.queue_time_us":83}
Sep 27 14:52:16 dist-test-slave-xk1g krb5kdc[8541](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 14:52:16 dist-test-slave-xk1g krb5kdc[8541](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 14:52:16.345629 9575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:52:16.337580 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39465 (local address 127.0.98.1:40705)
0927 14:52:16.337876 (+ 296us) negotiation.cc:107] Waiting for socket to connect
0927 14:52:16.337894 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 14:52:16.338011 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:52:16.338255 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:52:16.338257 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:52:16.338432 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0927 14:52:16.338520 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:52:16.338525 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:52:16.339230 (+ 705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:52:16.339232 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:52:16.339772 (+ 540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:52:16.339779 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:52:16.340173 (+ 394us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:52:16.340176 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:52:16.340236 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:52:16.343630 (+ 3394us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 14:52:16.343642 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 14:52:16.345500 (+ 1858us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:39465: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":197,"thread_start_us":102,"threads_started":1}
W20260927 14:52:16.345805 9576 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 14:52:16.337623 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:40705 (local address 127.0.98.62:39465)
0927 14:52:16.337919 (+ 296us) server_negotiation.cc:206] Beginning negotiation
0927 14:52:16.337922 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 14:52:16.337958 (+ 36us) server_negotiation.cc:407] Connection header received
0927 14:52:16.338098 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:52:16.338100 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:52:16.338146 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:52:16.338224 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:52:16.338611 (+ 387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:52:16.339143 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:52:16.339875 (+ 732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:52:16.340085 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:52:16.340157 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:52:16.345657 (+ 5500us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:40705: BlockingRecv error: recv got EOF from 127.0.98.1:40705 (error 108)
Metrics: {"server-negotiator.queue_time_us":234,"thread_start_us":73,"threads_started":1}
Sep 27 14:52:16 dist-test-slave-xk1g krb5kdc[8541](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 14:52:16 dist-test-slave-xk1g krb5kdc[8541](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 14:52:16.519570 9577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:52:16.514224 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39465 (local address 127.0.98.2:49691)
0927 14:52:16.514484 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0927 14:52:16.514498 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 14:52:16.514585 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:52:16.514911 (+ 326us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:52:16.514913 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:52:16.515084 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0927 14:52:16.515173 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:52:16.515179 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:52:16.515889 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:52:16.515891 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:52:16.516432 (+ 541us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:52:16.516439 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:52:16.516893 (+ 454us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:52:16.516895 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:52:16.516948 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:52:16.517429 (+ 481us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 14:52:16.517439 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0927 14:52:16.519460 (+ 2021us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:39465: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":83,"threads_started":1}
W20260927 14:52:16.519730 9576 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 14:52:16.514344 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:49691 (local address 127.0.98.62:39465)
0927 14:52:16.514535 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0927 14:52:16.514539 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 14:52:16.514554 (+ 15us) server_negotiation.cc:407] Connection header received
0927 14:52:16.514746 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:52:16.514749 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:52:16.514798 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:52:16.514874 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:52:16.515270 (+ 396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:52:16.515797 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:52:16.516562 (+ 765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:52:16.516794 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:52:16.516880 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:52:16.519582 (+ 2702us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:49691: BlockingRecv error: recv got EOF from 127.0.98.2:49691 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
Sep 27 14:52:16 dist-test-slave-xk1g krb5kdc[8541](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 14:52:16 dist-test-slave-xk1g krb5kdc[8541](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 14:52:16.708367 9578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:52:16.702963 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39465 (local address 127.0.98.3:53893)
0927 14:52:16.703229 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0927 14:52:16.703244 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 14:52:16.703343 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:52:16.703578 (+ 235us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:52:16.703580 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:52:16.703755 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0927 14:52:16.703846 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:52:16.703862 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:52:16.704669 (+ 807us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:52:16.704672 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:52:16.705381 (+ 709us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:52:16.705391 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:52:16.705860 (+ 469us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:52:16.705863 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:52:16.705949 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:52:16.706430 (+ 481us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 14:52:16.706440 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0927 14:52:16.708249 (+ 1809us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:39465: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":172,"thread_start_us":97,"threads_started":1}
W20260927 14:52:16.708500 9576 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 14:52:16.703006 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:53893 (local address 127.0.98.62:39465)
0927 14:52:16.703158 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0927 14:52:16.703162 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 14:52:16.703324 (+ 162us) server_negotiation.cc:407] Connection header received
0927 14:52:16.703430 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:52:16.703432 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:52:16.703479 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:52:16.703547 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:52:16.703948 (+ 401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:52:16.704579 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:52:16.705503 (+ 924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:52:16.705754 (+ 251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:52:16.705834 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:52:16.708382 (+ 2548us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:53893: BlockingRecv error: recv got EOF from 127.0.98.3:53893 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
Sep 27 14:52:17 dist-test-slave-xk1g krb5kdc[8541](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 14:52:17 dist-test-slave-xk1g krb5kdc[8541](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 14:52:17.351490 9579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:52:17.346463 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39465 (local address 127.0.98.1:43871)
0927 14:52:17.346715 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0927 14:52:17.346729 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 14:52:17.346812 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:52:17.347057 (+ 245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:52:17.347059 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:52:17.347232 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0927 14:52:17.347321 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:52:17.347326 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:52:17.348050 (+ 724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:52:17.348052 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:52:17.348595 (+ 543us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:52:17.348602 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:52:17.349120 (+ 518us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:52:17.349124 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:52:17.349184 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:52:17.349675 (+ 491us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 14:52:17.349686 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 14:52:17.351377 (+ 1691us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:39465: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":162,"thread_start_us":84,"threads_started":1}
W20260927 14:52:17.351640 9580 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 14:52:17.346526 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:43871 (local address 127.0.98.62:39465)
0927 14:52:17.346827 (+ 301us) server_negotiation.cc:206] Beginning negotiation
0927 14:52:17.346831 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 14:52:17.346862 (+ 31us) server_negotiation.cc:407] Connection header received
0927 14:52:17.346898 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:52:17.346900 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:52:17.346943 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:52:17.347027 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:52:17.347427 (+ 400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:52:17.347959 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:52:17.348721 (+ 762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:52:17.348963 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:52:17.349110 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:52:17.351507 (+ 2397us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:43871: BlockingRecv error: recv got EOF from 127.0.98.1:43871 (error 108)
Metrics: {"server-negotiator.queue_time_us":222,"thread_start_us":90,"threads_started":1}
Sep 27 14:52:17 dist-test-slave-xk1g krb5kdc[8541](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 14:52:17 dist-test-slave-xk1g krb5kdc[8541](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 14:52:17.526222 9581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:52:17.520492 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39465 (local address 127.0.98.2:37457)
0927 14:52:17.520803 (+ 311us) negotiation.cc:107] Waiting for socket to connect
0927 14:52:17.520816 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 14:52:17.520923 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:52:17.521213 (+ 290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:52:17.521216 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:52:17.521427 (+ 211us) client_negotiation.cc:189] Negotiated authn=SASL
0927 14:52:17.521515 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:52:17.521521 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:52:17.522257 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:52:17.522261 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:52:17.522871 (+ 610us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:52:17.522880 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:52:17.523319 (+ 439us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:52:17.523323 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:52:17.523428 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:52:17.524085 (+ 657us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 14:52:17.524096 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 14:52:17.526058 (+ 1962us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:39465: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":210,"thread_start_us":83,"threads_started":1}
W20260927 14:52:17.526276 9580 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 14:52:17.520611 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:37457 (local address 127.0.98.62:39465)
0927 14:52:17.520768 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0927 14:52:17.520773 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 14:52:17.520875 (+ 102us) server_negotiation.cc:407] Connection header received
0927 14:52:17.521033 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:52:17.521042 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:52:17.521105 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:52:17.521190 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:52:17.521629 (+ 439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:52:17.522152 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:52:17.522989 (+ 837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:52:17.523209 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:52:17.523318 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:52:17.526173 (+ 2855us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:37457: BlockingRecv error: recv got EOF from 127.0.98.2:37457 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Sep 27 14:52:17 dist-test-slave-xk1g krb5kdc[8541](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 14:52:17 dist-test-slave-xk1g krb5kdc[8541](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 14:52:17.714419 9582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:52:17.709212 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39465 (local address 127.0.98.3:59997)
0927 14:52:17.709636 (+ 424us) negotiation.cc:107] Waiting for socket to connect
0927 14:52:17.709651 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 14:52:17.709763 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:52:17.709993 (+ 230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:52:17.709995 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:52:17.710165 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0927 14:52:17.710252 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:52:17.710257 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:52:17.710954 (+ 697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:52:17.710956 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:52:17.711486 (+ 530us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:52:17.711492 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:52:17.711898 (+ 406us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:52:17.711900 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:52:17.711958 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:52:17.712459 (+ 501us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 14:52:17.712469 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0927 14:52:17.714308 (+ 1839us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:39465: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":333,"thread_start_us":83,"threads_started":1}
W20260927 14:52:17.714586 9580 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 14:52:17.709377 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:59997 (local address 127.0.98.62:39465)
0927 14:52:17.709499 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0927 14:52:17.709508 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 14:52:17.709731 (+ 223us) server_negotiation.cc:407] Connection header received
0927 14:52:17.709848 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:52:17.709850 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:52:17.709894 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:52:17.709959 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:52:17.710342 (+ 383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:52:17.710865 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:52:17.711572 (+ 707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:52:17.711804 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:52:17.711875 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:52:17.714433 (+ 2558us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:59997: BlockingRecv error: recv got EOF from 127.0.98.3:59997 (error 108)
Metrics: {"server-negotiator.queue_time_us":36}
Sep 27 14:52:18 dist-test-slave-xk1g krb5kdc[8541](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 14:52:18 dist-test-slave-xk1g krb5kdc[8541](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 14:52:18.358273 9583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:52:18.352545 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39465 (local address 127.0.98.1:38463)
0927 14:52:18.352892 (+ 347us) negotiation.cc:107] Waiting for socket to connect
0927 14:52:18.352908 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 14:52:18.353021 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:52:18.353381 (+ 360us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:52:18.353384 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:52:18.353633 (+ 249us) client_negotiation.cc:189] Negotiated authn=SASL
0927 14:52:18.353760 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:52:18.353768 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:52:18.354581 (+ 813us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:52:18.354583 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:52:18.355141 (+ 558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:52:18.355148 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:52:18.355603 (+ 455us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:52:18.355605 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:52:18.355666 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:52:18.356201 (+ 535us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 14:52:18.356212 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 14:52:18.358148 (+ 1936us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:39465: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":247,"thread_start_us":108,"threads_started":1}
W20260927 14:52:18.358528 9584 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 14:52:18.352666 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:38463 (local address 127.0.98.62:39465)
0927 14:52:18.352926 (+ 260us) server_negotiation.cc:206] Beginning negotiation
0927 14:52:18.352931 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 14:52:18.352967 (+ 36us) server_negotiation.cc:407] Connection header received
0927 14:52:18.353162 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:52:18.353165 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:52:18.353227 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:52:18.353327 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:52:18.353920 (+ 593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:52:18.354481 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:52:18.355265 (+ 784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:52:18.355498 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:52:18.355587 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:52:18.358385 (+ 2798us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:38463: BlockingRecv error: recv got EOF from 127.0.98.1:38463 (error 108)
Metrics: {"server-negotiator.queue_time_us":183,"thread_start_us":75,"threads_started":1}
Sep 27 14:52:18 dist-test-slave-xk1g krb5kdc[8541](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 14:52:18 dist-test-slave-xk1g krb5kdc[8541](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 14:52:18.532696 9585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:52:18.527183 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39465 (local address 127.0.98.2:53303)
0927 14:52:18.527538 (+ 355us) negotiation.cc:107] Waiting for socket to connect
0927 14:52:18.527551 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 14:52:18.527680 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:52:18.527978 (+ 298us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:52:18.527981 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:52:18.528209 (+ 228us) client_negotiation.cc:189] Negotiated authn=SASL
0927 14:52:18.528303 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:52:18.528309 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:52:18.529015 (+ 706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:52:18.529017 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:52:18.529635 (+ 618us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:52:18.529643 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:52:18.530130 (+ 487us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:52:18.530132 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:52:18.530196 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:52:18.530756 (+ 560us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 14:52:18.530767 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 14:52:18.532581 (+ 1814us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:39465: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":286,"thread_start_us":84,"threads_started":1}
W20260927 14:52:18.532887 9584 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 14:52:18.527221 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:53303 (local address 127.0.98.62:39465)
0927 14:52:18.527368 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0927 14:52:18.527374 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 14:52:18.527647 (+ 273us) server_negotiation.cc:407] Connection header received
0927 14:52:18.527805 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:52:18.527808 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:52:18.527865 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:52:18.527950 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:52:18.528404 (+ 454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:52:18.528926 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:52:18.529758 (+ 832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:52:18.530009 (+ 251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:52:18.530109 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:52:18.532731 (+ 2622us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:53303: BlockingRecv error: recv got EOF from 127.0.98.2:53303 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
Sep 27 14:52:18 dist-test-slave-xk1g krb5kdc[8541](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 14:52:18 dist-test-slave-xk1g krb5kdc[8541](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 14:52:18.719996 9586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:52:18.715186 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39465 (local address 127.0.98.3:46275)
0927 14:52:18.715436 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0927 14:52:18.715449 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 14:52:18.715538 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:52:18.715768 (+ 230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:52:18.715770 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:52:18.715959 (+ 189us) client_negotiation.cc:189] Negotiated authn=SASL
0927 14:52:18.716065 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:52:18.716070 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:52:18.716782 (+ 712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:52:18.716784 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:52:18.717333 (+ 549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:52:18.717347 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:52:18.717715 (+ 368us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:52:18.717718 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:52:18.717783 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:52:18.718240 (+ 457us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 14:52:18.718251 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 14:52:18.719890 (+ 1639us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:39465: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":91,"threads_started":1}
W20260927 14:52:18.720127 9584 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 14:52:18.715228 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:46275 (local address 127.0.98.62:39465)
0927 14:52:18.715362 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0927 14:52:18.715367 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 14:52:18.715513 (+ 146us) server_negotiation.cc:407] Connection header received
0927 14:52:18.715611 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:52:18.715613 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:52:18.715657 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:52:18.715739 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:52:18.716159 (+ 420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:52:18.716698 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:52:18.717425 (+ 727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:52:18.717626 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:52:18.717712 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:52:18.720018 (+ 2306us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:46275: BlockingRecv error: recv got EOF from 127.0.98.3:46275 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Sep 27 14:52:19 dist-test-slave-xk1g krb5kdc[8541](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 14:52:19 dist-test-slave-xk1g krb5kdc[8541](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 14:52:19.364589 9587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:52:19.359199 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39465 (local address 127.0.98.1:44495)
0927 14:52:19.359512 (+ 313us) negotiation.cc:107] Waiting for socket to connect
0927 14:52:19.359526 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 14:52:19.359619 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:52:19.359887 (+ 268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:52:19.359889 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:52:19.360062 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0927 14:52:19.360180 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:52:19.360191 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:52:19.360919 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:52:19.360922 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:52:19.361562 (+ 640us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:52:19.361572 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:52:19.362038 (+ 466us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:52:19.362042 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:52:19.362127 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:52:19.362673 (+ 546us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 14:52:19.362685 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 14:52:19.364479 (+ 1794us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:39465: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":234,"thread_start_us":97,"threads_started":1}
W20260927 14:52:19.364771 9588 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 14:52:19.359271 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:44495 (local address 127.0.98.62:39465)
0927 14:52:19.359557 (+ 286us) server_negotiation.cc:206] Beginning negotiation
0927 14:52:19.359561 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 14:52:19.359576 (+ 15us) server_negotiation.cc:407] Connection header received
0927 14:52:19.359707 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:52:19.359709 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:52:19.359769 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:52:19.359847 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:52:19.360289 (+ 442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:52:19.360817 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:52:19.361686 (+ 869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:52:19.361925 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:52:19.362036 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:52:19.364604 (+ 2568us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:44495: BlockingRecv error: recv got EOF from 127.0.98.1:44495 (error 108)
Metrics: {"server-negotiator.queue_time_us":218,"thread_start_us":104,"threads_started":1}
Sep 27 14:52:19 dist-test-slave-xk1g krb5kdc[8541](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 14:52:19 dist-test-slave-xk1g krb5kdc[8541](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 14:52:19.539269 9589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:52:19.533604 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39465 (local address 127.0.98.2:54317)
0927 14:52:19.533969 (+ 365us) negotiation.cc:107] Waiting for socket to connect
0927 14:52:19.533986 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 14:52:19.534111 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:52:19.534421 (+ 310us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:52:19.534424 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:52:19.534625 (+ 201us) client_negotiation.cc:189] Negotiated authn=SASL
0927 14:52:19.534747 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:52:19.534755 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:52:19.535538 (+ 783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:52:19.535541 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:52:19.536095 (+ 554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:52:19.536104 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:52:19.536580 (+ 476us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:52:19.536582 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:52:19.536645 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:52:19.537245 (+ 600us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 14:52:19.537256 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 14:52:19.539154 (+ 1898us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:39465: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":267,"thread_start_us":100,"threads_started":1}
W20260927 14:52:19.539449 9588 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 14:52:19.533721 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:54317 (local address 127.0.98.62:39465)
0927 14:52:19.533888 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0927 14:52:19.533893 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 14:52:19.534116 (+ 223us) server_negotiation.cc:407] Connection header received
0927 14:52:19.534227 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:52:19.534230 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:52:19.534288 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:52:19.534358 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:52:19.534879 (+ 521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:52:19.535438 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:52:19.536230 (+ 792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:52:19.536453 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:52:19.536565 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:52:19.539295 (+ 2730us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:54317: BlockingRecv error: recv got EOF from 127.0.98.2:54317 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Sep 27 14:52:19 dist-test-slave-xk1g krb5kdc[8541](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 14:52:19 dist-test-slave-xk1g krb5kdc[8541](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 14:52:19.725775 9590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:52:19.720816 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39465 (local address 127.0.98.3:44195)
0927 14:52:19.721112 (+ 296us) negotiation.cc:107] Waiting for socket to connect
0927 14:52:19.721125 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 14:52:19.721217 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:52:19.721434 (+ 217us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:52:19.721436 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:52:19.721604 (+ 168us) client_negotiation.cc:189] Negotiated authn=SASL
0927 14:52:19.721693 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:52:19.721697 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:52:19.722393 (+ 696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:52:19.722395 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:52:19.722943 (+ 548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:52:19.722950 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:52:19.723356 (+ 406us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:52:19.723358 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:52:19.723426 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:52:19.723965 (+ 539us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 14:52:19.723977 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 14:52:19.725666 (+ 1689us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:39465: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":207,"thread_start_us":117,"threads_started":1}
W20260927 14:52:19.725966 9588 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 14:52:19.720846 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:44195 (local address 127.0.98.62:39465)
0927 14:52:19.720991 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0927 14:52:19.720995 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 14:52:19.721190 (+ 195us) server_negotiation.cc:407] Connection header received
0927 14:52:19.721283 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:52:19.721285 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:52:19.721333 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:52:19.721398 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:52:19.721786 (+ 388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:52:19.722305 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:52:19.723040 (+ 735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:52:19.723260 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:52:19.723337 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:52:19.725831 (+ 2494us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:44195: BlockingRecv error: recv got EOF from 127.0.98.3:44195 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
Sep 27 14:52:20 dist-test-slave-xk1g krb5kdc[8541](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 14:52:20 dist-test-slave-xk1g krb5kdc[8541](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 14:52:20.370795 9591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:52:20.365413 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39465 (local address 127.0.98.1:51673)
0927 14:52:20.365683 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0927 14:52:20.365701 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 14:52:20.365808 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:52:20.366123 (+ 315us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:52:20.366126 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:52:20.366335 (+ 209us) client_negotiation.cc:189] Negotiated authn=SASL
0927 14:52:20.366424 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:52:20.366429 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:52:20.367185 (+ 756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:52:20.367189 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:52:20.367766 (+ 577us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:52:20.367775 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:52:20.368242 (+ 467us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:52:20.368246 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:52:20.368331 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:52:20.368839 (+ 508us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 14:52:20.368851 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 14:52:20.370672 (+ 1821us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:39465: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":175,"thread_start_us":66,"threads_started":1}
W20260927 14:52:20.370937 9592 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 14:52:20.365564 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:51673 (local address 127.0.98.62:39465)
0927 14:52:20.365848 (+ 284us) server_negotiation.cc:206] Beginning negotiation
0927 14:52:20.365854 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 14:52:20.365870 (+ 16us) server_negotiation.cc:407] Connection header received
0927 14:52:20.365914 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:52:20.365917 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:52:20.365973 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:52:20.366099 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:52:20.366539 (+ 440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:52:20.367073 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:52:20.367887 (+ 814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:52:20.368126 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:52:20.368238 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:52:20.370813 (+ 2575us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:51673: BlockingRecv error: recv got EOF from 127.0.98.1:51673 (error 108)
Metrics: {"server-negotiator.queue_time_us":144,"thread_start_us":71,"threads_started":1}
Sep 27 14:52:20 dist-test-slave-xk1g krb5kdc[8541](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 14:52:20 dist-test-slave-xk1g krb5kdc[8541](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 14:52:20.545410 9593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:52:20.540114 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39465 (local address 127.0.98.2:37001)
0927 14:52:20.540471 (+ 357us) negotiation.cc:107] Waiting for socket to connect
0927 14:52:20.540490 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 14:52:20.540611 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:52:20.540928 (+ 317us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:52:20.540932 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:52:20.541190 (+ 258us) client_negotiation.cc:189] Negotiated authn=SASL
0927 14:52:20.541313 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:52:20.541323 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:52:20.542050 (+ 727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:52:20.542053 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:52:20.542595 (+ 542us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:52:20.542601 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:52:20.543000 (+ 399us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:52:20.543002 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:52:20.543060 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:52:20.543530 (+ 470us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 14:52:20.543541 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 14:52:20.545294 (+ 1753us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:39465: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":251,"thread_start_us":90,"threads_started":1}
W20260927 14:52:20.545549 9592 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 14:52:20.540273 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:37001 (local address 127.0.98.62:39465)
0927 14:52:20.540458 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0927 14:52:20.540462 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 14:52:20.540594 (+ 132us) server_negotiation.cc:407] Connection header received
0927 14:52:20.540714 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:52:20.540717 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:52:20.540787 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:52:20.540866 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:52:20.541435 (+ 569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:52:20.541965 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:52:20.542687 (+ 722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:52:20.542909 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:52:20.542984 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:52:20.545435 (+ 2451us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:37001: BlockingRecv error: recv got EOF from 127.0.98.2:37001 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
Sep 27 14:52:20 dist-test-slave-xk1g krb5kdc[8541](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 14:52:20 dist-test-slave-xk1g krb5kdc[8541](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 14:52:20.731849 9594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:52:20.726505 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39465 (local address 127.0.98.3:49745)
0927 14:52:20.726810 (+ 305us) negotiation.cc:107] Waiting for socket to connect
0927 14:52:20.726825 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 14:52:20.726929 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:52:20.727206 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:52:20.727209 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:52:20.727430 (+ 221us) client_negotiation.cc:189] Negotiated authn=SASL
0927 14:52:20.727537 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:52:20.727542 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:52:20.728574 (+ 1032us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:52:20.728576 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:52:20.729164 (+ 588us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:52:20.729172 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:52:20.729563 (+ 391us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:52:20.729565 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:52:20.729623 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:52:20.730073 (+ 450us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 14:52:20.730084 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 14:52:20.731738 (+ 1654us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:39465: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":206,"thread_start_us":90,"threads_started":1}
W20260927 14:52:20.732018 9592 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 14:52:20.726569 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:49745 (local address 127.0.98.62:39465)
0927 14:52:20.726804 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0927 14:52:20.726808 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 14:52:20.726910 (+ 102us) server_negotiation.cc:407] Connection header received
0927 14:52:20.727014 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:52:20.727017 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:52:20.727072 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:52:20.727158 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:52:20.727648 (+ 490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:52:20.728459 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:52:20.729256 (+ 797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:52:20.729471 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:52:20.729548 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:52:20.731890 (+ 2342us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:49745: BlockingRecv error: recv got EOF from 127.0.98.3:49745 (error 108)
Metrics: {"server-negotiator.queue_time_us":135}
Sep 27 14:52:21 dist-test-slave-xk1g krb5kdc[8541](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 14:52:21 dist-test-slave-xk1g krb5kdc[8541](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 14:52:21.380031 9595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:52:21.371520 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39465 (local address 127.0.98.1:38451)
0927 14:52:21.371822 (+ 302us) negotiation.cc:107] Waiting for socket to connect
0927 14:52:21.371837 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 14:52:21.371922 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:52:21.372132 (+ 210us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:52:21.372134 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:52:21.372298 (+ 164us) client_negotiation.cc:189] Negotiated authn=SASL
0927 14:52:21.372395 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:52:21.372400 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:52:21.373161 (+ 761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:52:21.373163 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:52:21.373709 (+ 546us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:52:21.373716 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:52:21.374139 (+ 423us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:52:21.374141 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:52:21.374199 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:52:21.378073 (+ 3874us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 14:52:21.378084 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 14:52:21.379920 (+ 1836us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:39465: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":227,"thread_start_us":111,"threads_started":1}
W20260927 14:52:21.380290 9596 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 14:52:21.371578 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:38451 (local address 127.0.98.62:39465)
0927 14:52:21.371907 (+ 329us) server_negotiation.cc:206] Beginning negotiation
0927 14:52:21.371911 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 14:52:21.371923 (+ 12us) server_negotiation.cc:407] Connection header received
0927 14:52:21.371986 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:52:21.371989 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:52:21.372029 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:52:21.372102 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:52:21.372491 (+ 389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:52:21.373015 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:52:21.373804 (+ 789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:52:21.374016 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:52:21.374119 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:52:21.380103 (+ 5984us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:38451: BlockingRecv error: recv got EOF from 127.0.98.1:38451 (error 108)
Metrics: {"server-negotiator.queue_time_us":271,"thread_start_us":96,"threads_started":1}
Sep 27 14:52:21 dist-test-slave-xk1g krb5kdc[8541](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 14:52:21 dist-test-slave-xk1g krb5kdc[8541](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 14:52:21.551601 9597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:52:21.546282 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39465 (local address 127.0.98.2:41255)
0927 14:52:21.546537 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0927 14:52:21.546559 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0927 14:52:21.546687 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:52:21.547003 (+ 316us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:52:21.547006 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:52:21.547195 (+ 189us) client_negotiation.cc:189] Negotiated authn=SASL
0927 14:52:21.547315 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:52:21.547323 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:52:21.548179 (+ 856us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:52:21.548182 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:52:21.548725 (+ 543us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:52:21.548733 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:52:21.549182 (+ 449us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:52:21.549184 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:52:21.549246 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:52:21.549688 (+ 442us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 14:52:21.549698 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0927 14:52:21.551467 (+ 1769us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:39465: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":92,"threads_started":1}
W20260927 14:52:21.551743 9596 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 14:52:21.546361 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:41255 (local address 127.0.98.62:39465)
0927 14:52:21.546503 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0927 14:52:21.546509 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 14:52:21.546634 (+ 125us) server_negotiation.cc:407] Connection header received
0927 14:52:21.546811 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:52:21.546817 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:52:21.546883 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:52:21.546970 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:52:21.547436 (+ 466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:52:21.548080 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:52:21.548823 (+ 743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:52:21.549041 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:52:21.549183 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:52:21.551599 (+ 2416us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:41255: BlockingRecv error: recv got EOF from 127.0.98.2:41255 (error 108)
Metrics: {"server-negotiator.queue_time_us":33}
Sep 27 14:52:21 dist-test-slave-xk1g krb5kdc[8541](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 14:52:21 dist-test-slave-xk1g krb5kdc[8541](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 14:52:21.738328 9598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:52:21.732625 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39465 (local address 127.0.98.3:43627)
0927 14:52:21.733038 (+ 413us) negotiation.cc:107] Waiting for socket to connect
0927 14:52:21.733082 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0927 14:52:21.733192 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:52:21.733437 (+ 245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:52:21.733439 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:52:21.733609 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0927 14:52:21.733699 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:52:21.733704 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:52:21.734398 (+ 694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:52:21.734400 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:52:21.734942 (+ 542us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:52:21.734949 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:52:21.735354 (+ 405us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:52:21.735356 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:52:21.735411 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:52:21.735905 (+ 494us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 14:52:21.735916 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 14:52:21.738156 (+ 2240us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:39465: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":78,"threads_started":1}
W20260927 14:52:21.738368 9596 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 14:52:21.732724 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:43627 (local address 127.0.98.62:39465)
0927 14:52:21.732902 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0927 14:52:21.732905 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 14:52:21.733160 (+ 255us) server_negotiation.cc:407] Connection header received
0927 14:52:21.733273 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:52:21.733275 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:52:21.733319 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:52:21.733386 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:52:21.733804 (+ 418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:52:21.734311 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:52:21.735045 (+ 734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:52:21.735260 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:52:21.735339 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:52:21.738274 (+ 2935us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:43627: BlockingRecv error: recv got EOF from 127.0.98.3:43627 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
Sep 27 14:52:22 dist-test-slave-xk1g krb5kdc[8541](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 14:52:22 dist-test-slave-xk1g krb5kdc[8541](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 14:52:22.386844 9599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:52:22.381031 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39465 (local address 127.0.98.1:50779)
0927 14:52:22.381328 (+ 297us) negotiation.cc:107] Waiting for socket to connect
0927 14:52:22.381342 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 14:52:22.381429 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:52:22.381735 (+ 306us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:52:22.381738 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:52:22.381926 (+ 188us) client_negotiation.cc:189] Negotiated authn=SASL
0927 14:52:22.382021 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:52:22.382026 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:52:22.382798 (+ 772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:52:22.382801 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:52:22.383558 (+ 757us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:52:22.383567 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:52:22.384090 (+ 523us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:52:22.384095 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:52:22.384202 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:52:22.384781 (+ 579us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 14:52:22.384795 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0927 14:52:22.386714 (+ 1919us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:39465: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":220,"thread_start_us":80,"threads_started":1}
W20260927 14:52:22.387007 9600 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 14:52:22.381185 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:50779 (local address 127.0.98.62:39465)
0927 14:52:22.381410 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0927 14:52:22.381414 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 14:52:22.381428 (+ 14us) server_negotiation.cc:407] Connection header received
0927 14:52:22.381527 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:52:22.381530 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:52:22.381571 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:52:22.381651 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:52:22.382131 (+ 480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:52:22.382689 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:52:22.383670 (+ 981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:52:22.383970 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:52:22.384054 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:52:22.386844 (+ 2790us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:50779: BlockingRecv error: recv got EOF from 127.0.98.1:50779 (error 108)
Metrics: {"server-negotiator.queue_time_us":163,"thread_start_us":57,"threads_started":1}
Sep 27 14:52:22 dist-test-slave-xk1g krb5kdc[8541](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 14:52:22 dist-test-slave-xk1g krb5kdc[8541](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 14:52:22.558308 9601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:52:22.552523 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39465 (local address 127.0.98.2:45415)
0927 14:52:22.552834 (+ 311us) negotiation.cc:107] Waiting for socket to connect
0927 14:52:22.552850 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 14:52:22.552969 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:52:22.553315 (+ 346us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:52:22.553332 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:52:22.553564 (+ 232us) client_negotiation.cc:189] Negotiated authn=SASL
0927 14:52:22.553686 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:52:22.553693 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:52:22.554600 (+ 907us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:52:22.554603 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:52:22.555158 (+ 555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:52:22.555165 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:52:22.555563 (+ 398us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:52:22.555566 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:52:22.555626 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:52:22.556176 (+ 550us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 14:52:22.556190 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0927 14:52:22.558163 (+ 1973us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:39465: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":209,"thread_start_us":84,"threads_started":1}
W20260927 14:52:22.558379 9600 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 14:52:22.552646 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:45415 (local address 127.0.98.62:39465)
0927 14:52:22.552835 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0927 14:52:22.552841 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 14:52:22.552940 (+ 99us) server_negotiation.cc:407] Connection header received
0927 14:52:22.553126 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:52:22.553129 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:52:22.553188 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:52:22.553281 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:52:22.553822 (+ 541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:52:22.554504 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:52:22.555264 (+ 760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:52:22.555472 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:52:22.555542 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:52:22.558288 (+ 2746us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:45415: BlockingRecv error: recv got EOF from 127.0.98.2:45415 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
I20260927 14:52:22.644989 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6936m5/build/debug/bin/kudu with pid 9124
I20260927 14:52:22.650606 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6936m5/build/debug/bin/kudu with pid 9276
I20260927 14:52:22.655759 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6936m5/build/debug/bin/kudu with pid 9408
I20260927 14:52:22.661020 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6936m5/build/debug/bin/kudu with pid 9054
2026-09-27T14:52:22Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (17941 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-09-27T14:52:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T14:52:22Z Disabled control of system clock
I20260927 14:52:22.689709 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6936m5/build/debug/bin/kudu
/tmp/dist-test-task6936m5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790520684602251-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790520684602251-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790520684602251-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790520684602251-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41879
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:35481
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:41879
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260927 14:52:22.797142 9605 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 14:52:22.797382 9605 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 14:52:22.797441 9605 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 14:52:22.801360 9605 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 14:52:22.801415 9605 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 14:52:22.801431 9605 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 14:52:22.801448 9605 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 14:52:22.801463 9605 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 14:52:22.806680 9605 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35481
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790520684602251-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790520684602251-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:41879
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41879
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790520684602251-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9605
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790520684602251-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:15:17 UTC on ccycloud.quasar-ledxfe.root.comops.site
build id 4676134
I20260927 14:52:22.808045 9605 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 14:52:22.809666 9605 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 14:52:22.817842 9610 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 14:52:22.817844 9611 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 14:52:22.818041 9605 server_base.cc:1034] running on GCE node
W20260927 14:52:22.817844 9613 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 14:52:22.818817 9605 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 14:52:22.819888 9605 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 14:52:22.821048 9605 hybrid_clock.cc:648] HybridClock initialized: now 1790520742821017 us; error 40 us; skew 500 ppm
I20260927 14:52:22.822818 9605 webserver.cc:460] Webserver started at http://127.0.98.62:37655/ using document root <none> and password file <none>
I20260927 14:52:22.823408 9605 fs_manager.cc:359] Metadata directory not provided
I20260927 14:52:22.823470 9605 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 14:52:22.823698 9605 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 14:52:22.825886 9605 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790520684602251-392-0/minicluster-data/master-0/data/instance:
uuid: "0d717fcccb8a4241b5bebec3c2f776de"
format_stamp: "Formatted at 2026-09-27 14:52:22 on dist-test-slave-xk1g"
server_key: "e749fc29039f20b31dc19d13ad66bfa1"
server_key_iv: "0bc1b340c7bdfd67e6091def32034165"
server_key_version: "encryptionkey@0"
I20260927 14:52:22.826448 9605 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790520684602251-392-0/minicluster-data/master-0/wal/instance:
uuid: "0d717fcccb8a4241b5bebec3c2f776de"
format_stamp: "Formatted at 2026-09-27 14:52:22 on dist-test-slave-xk1g"
server_key: "e749fc29039f20b31dc19d13ad66bfa1"
server_key_iv: "0bc1b340c7bdfd67e6091def32034165"
server_key_version: "encryptionkey@0"
I20260927 14:52:22.829946 9605 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 14:52:22.832291 9619 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 14:52:22.833540 9605 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 14:52:22.833678 9605 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790520684602251-392-0/minicluster-data/master-0/data,/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790520684602251-392-0/minicluster-data/master-0/wal
uuid: "0d717fcccb8a4241b5bebec3c2f776de"
format_stamp: "Formatted at 2026-09-27 14:52:22 on dist-test-slave-xk1g"
server_key: "e749fc29039f20b31dc19d13ad66bfa1"
server_key_iv: "0bc1b340c7bdfd67e6091def32034165"
server_key_version: "encryptionkey@0"
I20260927 14:52:22.833791 9605 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790520684602251-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790520684602251-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790520684602251-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 14:52:22.863165 9605 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 14:52:22.863911 9605 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 14:52:22.864118 9605 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 14:52:22.870951 9605 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:41879
I20260927 14:52:22.870962 9671 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:41879 every 8 connection(s)
I20260927 14:52:22.871903 9605 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790520684602251-392-0/minicluster-data/master-0/data/info.pb
I20260927 14:52:22.875001 9672 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 14:52:22.875100 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6936m5/build/debug/bin/kudu as pid 9605
I20260927 14:52:22.875200 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790520684602251-392-0/minicluster-data/master-0/wal/instance
I20260927 14:52:22.875509 392 external_mini_cluster.cc:1428] Setting key cd63d60329b50a9937ebb739874c958b
I20260927 14:52:22.881039 9672 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0d717fcccb8a4241b5bebec3c2f776de: Bootstrap starting.
I20260927 14:52:22.883309 9672 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0d717fcccb8a4241b5bebec3c2f776de: Neither blocks nor log segments found. Creating new log.
I20260927 14:52:22.884038 9672 log.cc:824] T 00000000000000000000000000000000 P 0d717fcccb8a4241b5bebec3c2f776de: Log is configured to *not* fsync() on all Append() calls
I20260927 14:52:22.885186 9675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:52:22.876931 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33080 (local address 127.0.98.62:41879)
0927 14:52:22.877566 (+ 635us) server_negotiation.cc:206] Beginning negotiation
0927 14:52:22.877582 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0927 14:52:22.877642 (+ 60us) server_negotiation.cc:407] Connection header received
0927 14:52:22.878299 (+ 657us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:52:22.878316 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:52:22.878705 (+ 389us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:52:22.879059 (+ 354us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:52:22.879479 (+ 420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:52:22.880460 (+ 981us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:52:22.881546 (+ 1086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:52:22.882020 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:52:22.882203 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:52:22.883118 (+ 915us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:52:22.883122 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:52:22.883135 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 14:52:22.883586 (+ 451us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:52:22.883641 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 14:52:22.883661 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:52:22.883928 (+ 267us) server_negotiation.cc:1035] Waiting for connection context
0927 14:52:22.884055 (+ 127us) server_negotiation.cc:299] Negotiation successful
0927 14:52:22.884423 (+ 368us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":293,"thread_start_us":124,"threads_started":1}
I20260927 14:52:22.885916 9672 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0d717fcccb8a4241b5bebec3c2f776de: No bootstrap required, opened a new log
I20260927 14:52:22.889259 9672 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0d717fcccb8a4241b5bebec3c2f776de [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d717fcccb8a4241b5bebec3c2f776de" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41879 } }
I20260927 14:52:22.889470 9672 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0d717fcccb8a4241b5bebec3c2f776de [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 14:52:22.889550 9672 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0d717fcccb8a4241b5bebec3c2f776de [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d717fcccb8a4241b5bebec3c2f776de, State: Initialized, Role: FOLLOWER
I20260927 14:52:22.889959 9672 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0d717fcccb8a4241b5bebec3c2f776de [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d717fcccb8a4241b5bebec3c2f776de" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41879 } }
I20260927 14:52:22.890096 9672 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0d717fcccb8a4241b5bebec3c2f776de [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 14:52:22.890177 9672 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0d717fcccb8a4241b5bebec3c2f776de [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 14:52:22.890286 9672 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0d717fcccb8a4241b5bebec3c2f776de [term 0 FOLLOWER]: Advancing to term 1
I20260927 14:52:22.891050 9672 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0d717fcccb8a4241b5bebec3c2f776de [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d717fcccb8a4241b5bebec3c2f776de" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41879 } }
I20260927 14:52:22.891376 9672 leader_election.cc:302] T 00000000000000000000000000000000 P 0d717fcccb8a4241b5bebec3c2f776de [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0d717fcccb8a4241b5bebec3c2f776de; no voters:
I20260927 14:52:22.891629 9672 leader_election.cc:288] T 00000000000000000000000000000000 P 0d717fcccb8a4241b5bebec3c2f776de [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 14:52:22.891758 9677 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0d717fcccb8a4241b5bebec3c2f776de [term 1 FOLLOWER]: Leader election won for term 1
I20260927 14:52:22.891996 9677 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0d717fcccb8a4241b5bebec3c2f776de [term 1 LEADER]: Becoming Leader. State: Replica: 0d717fcccb8a4241b5bebec3c2f776de, State: Running, Role: LEADER
I20260927 14:52:22.892319 9677 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0d717fcccb8a4241b5bebec3c2f776de [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d717fcccb8a4241b5bebec3c2f776de" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41879 } }
I20260927 14:52:22.892841 9672 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0d717fcccb8a4241b5bebec3c2f776de [sys.catalog]: configured and running, proceeding with master startup.
I20260927 14:52:22.893955 9678 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0d717fcccb8a4241b5bebec3c2f776de [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0d717fcccb8a4241b5bebec3c2f776de" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d717fcccb8a4241b5bebec3c2f776de" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41879 } } }
I20260927 14:52:22.894064 9678 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0d717fcccb8a4241b5bebec3c2f776de [sys.catalog]: This master's current role is: LEADER
I20260927 14:52:22.894440 9686 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 14:52:22.894462 9679 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0d717fcccb8a4241b5bebec3c2f776de [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0d717fcccb8a4241b5bebec3c2f776de. Latest consensus state: current_term: 1 leader_uuid: "0d717fcccb8a4241b5bebec3c2f776de" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d717fcccb8a4241b5bebec3c2f776de" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41879 } } }
I20260927 14:52:22.895107 9679 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0d717fcccb8a4241b5bebec3c2f776de [sys.catalog]: This master's current role is: LEADER
I20260927 14:52:22.897419 9686 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 14:52:22.902482 9686 catalog_manager.cc:1347] Generated new cluster ID: 8d76c70457914657b2c4ddbf90a6063a
I20260927 14:52:22.902546 9686 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 14:52:22.911872 9686 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 14:52:22.912705 9686 catalog_manager.cc:1504] Loading token signing keys...
I20260927 14:52:22.918547 9686 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0d717fcccb8a4241b5bebec3c2f776de: Generated new TSK 0
I20260927 14:52:22.919175 9686 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 14:52:22.923725 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6936m5/build/debug/bin/kudu
/tmp/dist-test-task6936m5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790520684602251-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790520684602251-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790520684602251-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790520684602251-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41879
--builtin_ntp_servers=127.0.98.20:35481
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260927 14:52:23.032189 9696 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 14:52:23.032444 9696 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 14:52:23.032480 9696 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 14:52:23.036473 9696 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 14:52:23.036535 9696 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 14:52:23.036588 9696 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 14:52:23.041473 9696 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35481
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790520684602251-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790520684602251-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790520684602251-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41879
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9696
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790520684602251-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:15:17 UTC on ccycloud.quasar-ledxfe.root.comops.site
build id 4676134
I20260927 14:52:23.042481 9696 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 14:52:23.043778 9696 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 14:52:23.052632 9702 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 14:52:23.052618 9701 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 14:52:23.052759 9696 server_base.cc:1034] running on GCE node
W20260927 14:52:23.052639 9704 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 14:52:23.053359 9696 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 14:52:23.053975 9696 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 14:52:23.055128 9696 hybrid_clock.cc:648] HybridClock initialized: now 1790520743055116 us; error 27 us; skew 500 ppm
I20260927 14:52:23.056770 9696 webserver.cc:460] Webserver started at http://127.0.98.1:42887/ using document root <none> and password file <none>
I20260927 14:52:23.057353 9696 fs_manager.cc:359] Metadata directory not provided
I20260927 14:52:23.057411 9696 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 14:52:23.057654 9696 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 14:52:23.059545 9696 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790520684602251-392-0/minicluster-data/ts-0/data/instance:
uuid: "8a3fb531a112466d9ac77966324861dd"
format_stamp: "Formatted at 2026-09-27 14:52:23 on dist-test-slave-xk1g"
server_key: "d45ede5e95932e7c95879ff94b918f80"
server_key_iv: "ed087090715b7ba68bb93db6ba7a4a21"
server_key_version: "encryptionkey@0"
I20260927 14:52:23.060070 9696 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790520684602251-392-0/minicluster-data/ts-0/wal/instance:
uuid: "8a3fb531a112466d9ac77966324861dd"
format_stamp: "Formatted at 2026-09-27 14:52:23 on dist-test-slave-xk1g"
server_key: "d45ede5e95932e7c95879ff94b918f80"
server_key_iv: "ed087090715b7ba68bb93db6ba7a4a21"
server_key_version: "encryptionkey@0"
I20260927 14:52:23.063493 9696 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260927 14:52:23.065888 9710 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 14:52:23.067008 9696 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.003s
I20260927 14:52:23.067147 9696 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790520684602251-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790520684602251-392-0/minicluster-data/ts-0/wal
uuid: "8a3fb531a112466d9ac77966324861dd"
format_stamp: "Formatted at 2026-09-27 14:52:23 on dist-test-slave-xk1g"
server_key: "d45ede5e95932e7c95879ff94b918f80"
server_key_iv: "ed087090715b7ba68bb93db6ba7a4a21"
server_key_version: "encryptionkey@0"
I20260927 14:52:23.067260 9696 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790520684602251-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790520684602251-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790520684602251-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 14:52:23.086791 9696 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 14:52:23.087440 9696 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 14:52:23.087625 9696 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 14:52:23.088181 9696 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 14:52:23.089116 9696 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 14:52:23.089184 9696 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 14:52:23.089252 9696 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 14:52:23.089301 9696 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 14:52:23.097643 9696 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:41675
I20260927 14:52:23.097663 9823 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:41675 every 8 connection(s)
I20260927 14:52:23.098493 9696 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790520684602251-392-0/minicluster-data/ts-0/data/info.pb
I20260927 14:52:23.099435 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6936m5/build/debug/bin/kudu as pid 9696
I20260927 14:52:23.099539 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790520684602251-392-0/minicluster-data/ts-0/wal/instance
I20260927 14:52:23.099825 392 external_mini_cluster.cc:1428] Setting key fe74f474bfb90456bfadb5d361bba5aa
I20260927 14:52:23.101752 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6936m5/build/debug/bin/kudu
/tmp/dist-test-task6936m5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790520684602251-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790520684602251-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790520684602251-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790520684602251-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41879
--builtin_ntp_servers=127.0.98.20:35481
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260927 14:52:23.107785 9675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:52:23.100400 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:60777 (local address 127.0.98.62:41879)
0927 14:52:23.100538 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0927 14:52:23.100542 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 14:52:23.101720 (+ 1178us) server_negotiation.cc:407] Connection header received
0927 14:52:23.103062 (+ 1342us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:52:23.103067 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:52:23.103129 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:52:23.103242 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:52:23.104058 (+ 816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:52:23.104611 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:52:23.105868 (+ 1257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:52:23.106096 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:52:23.106209 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:52:23.107294 (+ 1085us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:52:23.107302 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:52:23.107304 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 14:52:23.107315 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:52:23.107340 (+ 25us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 14:52:23.107349 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:52:23.107423 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0927 14:52:23.107582 (+ 159us) server_negotiation.cc:299] Negotiation successful
0927 14:52:23.107645 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260927 14:52:23.108579 9826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:52:23.100806 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41879 (local address 127.0.98.1:60777)
0927 14:52:23.101546 (+ 740us) negotiation.cc:107] Waiting for socket to connect
0927 14:52:23.101600 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0927 14:52:23.102885 (+ 1285us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:52:23.103393 (+ 508us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:52:23.103399 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:52:23.103477 (+ 78us) client_negotiation.cc:189] Negotiated authn=SASL
0927 14:52:23.103934 (+ 457us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:52:23.103946 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:52:23.104767 (+ 821us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:52:23.104771 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:52:23.105741 (+ 970us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:52:23.105751 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:52:23.106250 (+ 499us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:52:23.106254 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:52:23.106418 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:52:23.107045 (+ 627us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 14:52:23.107047 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0927 14:52:23.107084 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 14:52:23.107144 (+ 60us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 14:52:23.107437 (+ 293us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 14:52:23.107446 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 14:52:23.107451 (+ 5us) client_negotiation.cc:769] Sending connection context
0927 14:52:23.107570 (+ 119us) client_negotiation.cc:240] Negotiation successful
0927 14:52:23.107841 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":315,"thread_start_us":162,"threads_started":1}
I20260927 14:52:23.109896 9824 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41879
I20260927 14:52:23.110110 9824 heartbeater.cc:464] Registering TS with master...
I20260927 14:52:23.110571 9824 heartbeater.cc:511] Master 127.0.98.62:41879 requested a full tablet report, sending...
I20260927 14:52:23.111990 9636 ts_manager.cc:194] Registered new tserver with Master: 8a3fb531a112466d9ac77966324861dd (127.0.98.1:41675)
I20260927 14:52:23.113138 9636 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.1:60777
W20260927 14:52:23.208935 9827 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 14:52:23.209249 9827 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 14:52:23.209321 9827 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 14:52:23.213229 9827 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 14:52:23.213301 9827 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 14:52:23.213392 9827 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 14:52:23.217989 9827 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35481
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790520684602251-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790520684602251-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790520684602251-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41879
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9827
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790520684602251-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:15:17 UTC on ccycloud.quasar-ledxfe.root.comops.site
build id 4676134
I20260927 14:52:23.218938 9827 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 14:52:23.220197 9827 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 14:52:23.229116 9832 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 14:52:23.229103 9833 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 14:52:23.229092 9835 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 14:52:23.229252 9827 server_base.cc:1034] running on GCE node
I20260927 14:52:23.229759 9827 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 14:52:23.230329 9827 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 14:52:23.231519 9827 hybrid_clock.cc:648] HybridClock initialized: now 1790520743231483 us; error 44 us; skew 500 ppm
I20260927 14:52:23.233385 9827 webserver.cc:460] Webserver started at http://127.0.98.2:40445/ using document root <none> and password file <none>
I20260927 14:52:23.233916 9827 fs_manager.cc:359] Metadata directory not provided
I20260927 14:52:23.234000 9827 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 14:52:23.234231 9827 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 14:52:23.236145 9827 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790520684602251-392-0/minicluster-data/ts-1/data/instance:
uuid: "f43c02fac59249a98ab6ec1223857a47"
format_stamp: "Formatted at 2026-09-27 14:52:23 on dist-test-slave-xk1g"
server_key: "3f182507453628488522d36029091d8e"
server_key_iv: "a488fbbced4362942d9a1d1e794ba61f"
server_key_version: "encryptionkey@0"
I20260927 14:52:23.236754 9827 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790520684602251-392-0/minicluster-data/ts-1/wal/instance:
uuid: "f43c02fac59249a98ab6ec1223857a47"
format_stamp: "Formatted at 2026-09-27 14:52:23 on dist-test-slave-xk1g"
server_key: "3f182507453628488522d36029091d8e"
server_key_iv: "a488fbbced4362942d9a1d1e794ba61f"
server_key_version: "encryptionkey@0"
I20260927 14:52:23.240229 9827 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260927 14:52:23.242550 9841 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 14:52:23.243757 9827 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 14:52:23.243904 9827 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790520684602251-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790520684602251-392-0/minicluster-data/ts-1/wal
uuid: "f43c02fac59249a98ab6ec1223857a47"
format_stamp: "Formatted at 2026-09-27 14:52:23 on dist-test-slave-xk1g"
server_key: "3f182507453628488522d36029091d8e"
server_key_iv: "a488fbbced4362942d9a1d1e794ba61f"
server_key_version: "encryptionkey@0"
I20260927 14:52:23.244014 9827 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790520684602251-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790520684602251-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790520684602251-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 14:52:23.273262 9827 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 14:52:23.273999 9827 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 14:52:23.274200 9827 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 14:52:23.274798 9827 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 14:52:23.275732 9827 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 14:52:23.275820 9827 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 14:52:23.275890 9827 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 14:52:23.275938 9827 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 14:52:23.284366 9827 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:41785
I20260927 14:52:23.284394 9954 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:41785 every 8 connection(s)
I20260927 14:52:23.285292 9827 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790520684602251-392-0/minicluster-data/ts-1/data/info.pb
I20260927 14:52:23.287515 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6936m5/build/debug/bin/kudu as pid 9827
I20260927 14:52:23.287632 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790520684602251-392-0/minicluster-data/ts-1/wal/instance
I20260927 14:52:23.287901 392 external_mini_cluster.cc:1428] Setting key 15320f2d6f1c0262af08f94a032337a4
I20260927 14:52:23.289781 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6936m5/build/debug/bin/kudu
/tmp/dist-test-task6936m5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790520684602251-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790520684602251-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790520684602251-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790520684602251-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41879
--builtin_ntp_servers=127.0.98.20:35481
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260927 14:52:23.295154 9675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:52:23.286945 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:34651 (local address 127.0.98.62:41879)
0927 14:52:23.287082 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0927 14:52:23.287086 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 14:52:23.288195 (+ 1109us) server_negotiation.cc:407] Connection header received
0927 14:52:23.289721 (+ 1526us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:52:23.289725 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:52:23.289775 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:52:23.289886 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:52:23.290813 (+ 927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:52:23.291446 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:52:23.292869 (+ 1423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:52:23.293317 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:52:23.293445 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:52:23.294552 (+ 1107us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:52:23.294559 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:52:23.294566 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 14:52:23.294583 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:52:23.294612 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 14:52:23.294625 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:52:23.294733 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0927 14:52:23.294993 (+ 260us) server_negotiation.cc:299] Negotiation successful
0927 14:52:23.295045 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260927 14:52:23.296373 9957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:52:23.287344 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41879 (local address 127.0.98.2:34651)
0927 14:52:23.288029 (+ 685us) negotiation.cc:107] Waiting for socket to connect
0927 14:52:23.288078 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0927 14:52:23.289514 (+ 1436us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:52:23.290113 (+ 599us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:52:23.290119 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:52:23.290198 (+ 79us) client_negotiation.cc:189] Negotiated authn=SASL
0927 14:52:23.290663 (+ 465us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:52:23.290675 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:52:23.291571 (+ 896us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:52:23.291578 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:52:23.292741 (+ 1163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:52:23.292750 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:52:23.293454 (+ 704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:52:23.293456 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:52:23.293663 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:52:23.294306 (+ 643us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 14:52:23.294307 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0927 14:52:23.294347 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 14:52:23.294406 (+ 59us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 14:52:23.294785 (+ 379us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 14:52:23.294796 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 14:52:23.294805 (+ 9us) client_negotiation.cc:769] Sending connection context
0927 14:52:23.294981 (+ 176us) client_negotiation.cc:240] Negotiation successful
0927 14:52:23.295298 (+ 317us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":250,"thread_start_us":129,"threads_started":1}
I20260927 14:52:23.297531 9955 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41879
I20260927 14:52:23.297837 9955 heartbeater.cc:464] Registering TS with master...
I20260927 14:52:23.298332 9955 heartbeater.cc:511] Master 127.0.98.62:41879 requested a full tablet report, sending...
I20260927 14:52:23.299343 9636 ts_manager.cc:194] Registered new tserver with Master: f43c02fac59249a98ab6ec1223857a47 (127.0.98.2:41785)
I20260927 14:52:23.299893 9636 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.2:34651
W20260927 14:52:23.398488 9958 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 14:52:23.398756 9958 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 14:52:23.398828 9958 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 14:52:23.402737 9958 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 14:52:23.402820 9958 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 14:52:23.402911 9958 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 14:52:23.407513 9958 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35481
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790520684602251-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790520684602251-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790520684602251-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41879
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9958
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790520684602251-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:15:17 UTC on ccycloud.quasar-ledxfe.root.comops.site
build id 4676134
I20260927 14:52:23.408568 9958 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 14:52:23.409969 9958 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 14:52:23.419386 9966 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 14:52:23.419427 9963 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 14:52:23.419655 9958 server_base.cc:1034] running on GCE node
W20260927 14:52:23.419397 9964 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 14:52:23.420090 9958 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 14:52:23.420672 9958 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 14:52:23.421825 9958 hybrid_clock.cc:648] HybridClock initialized: now 1790520743421806 us; error 26 us; skew 500 ppm
I20260927 14:52:23.423493 9958 webserver.cc:460] Webserver started at http://127.0.98.3:33513/ using document root <none> and password file <none>
I20260927 14:52:23.424060 9958 fs_manager.cc:359] Metadata directory not provided
I20260927 14:52:23.424117 9958 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 14:52:23.424346 9958 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 14:52:23.426362 9958 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790520684602251-392-0/minicluster-data/ts-2/data/instance:
uuid: "702f62a037f6412888ef10938d60b553"
format_stamp: "Formatted at 2026-09-27 14:52:23 on dist-test-slave-xk1g"
server_key: "c08334b75d8d9c4f443d338a8d16ec53"
server_key_iv: "86c810df411846c7bdc8ca55c82718e1"
server_key_version: "encryptionkey@0"
I20260927 14:52:23.426900 9958 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790520684602251-392-0/minicluster-data/ts-2/wal/instance:
uuid: "702f62a037f6412888ef10938d60b553"
format_stamp: "Formatted at 2026-09-27 14:52:23 on dist-test-slave-xk1g"
server_key: "c08334b75d8d9c4f443d338a8d16ec53"
server_key_iv: "86c810df411846c7bdc8ca55c82718e1"
server_key_version: "encryptionkey@0"
I20260927 14:52:23.430301 9958 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 14:52:23.432562 9972 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 14:52:23.433794 9958 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 14:52:23.433923 9958 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790520684602251-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790520684602251-392-0/minicluster-data/ts-2/wal
uuid: "702f62a037f6412888ef10938d60b553"
format_stamp: "Formatted at 2026-09-27 14:52:23 on dist-test-slave-xk1g"
server_key: "c08334b75d8d9c4f443d338a8d16ec53"
server_key_iv: "86c810df411846c7bdc8ca55c82718e1"
server_key_version: "encryptionkey@0"
I20260927 14:52:23.434024 9958 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790520684602251-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790520684602251-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790520684602251-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 14:52:23.451120 9958 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 14:52:23.451761 9958 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 14:52:23.451951 9958 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 14:52:23.452500 9958 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 14:52:23.453440 9958 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 14:52:23.453509 9958 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 14:52:23.453579 9958 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 14:52:23.453630 9958 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 14:52:23.462005 9958 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:32853
I20260927 14:52:23.462031 10085 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:32853 every 8 connection(s)
I20260927 14:52:23.462895 9958 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790520684602251-392-0/minicluster-data/ts-2/data/info.pb
I20260927 14:52:23.465788 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6936m5/build/debug/bin/kudu as pid 9958
I20260927 14:52:23.465901 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790520684602251-392-0/minicluster-data/ts-2/wal/instance
I20260927 14:52:23.466147 392 external_mini_cluster.cc:1428] Setting key eaa91e9d77a7b6656e1719a0a73cc679
I20260927 14:52:23.472198 9675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:52:23.464647 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:58841 (local address 127.0.98.62:41879)
0927 14:52:23.464785 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0927 14:52:23.464788 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 14:52:23.465871 (+ 1083us) server_negotiation.cc:407] Connection header received
0927 14:52:23.467654 (+ 1783us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:52:23.467657 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:52:23.467698 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:52:23.467762 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:52:23.468750 (+ 988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:52:23.469327 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:52:23.470475 (+ 1148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:52:23.470678 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:52:23.470752 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:52:23.471828 (+ 1076us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:52:23.471831 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:52:23.471833 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 14:52:23.471844 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:52:23.471860 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 14:52:23.471867 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:52:23.471923 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0927 14:52:23.472051 (+ 128us) server_negotiation.cc:299] Negotiation successful
0927 14:52:23.472095 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260927 14:52:23.472995 10088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:52:23.465039 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41879 (local address 127.0.98.3:58841)
0927 14:52:23.465708 (+ 669us) negotiation.cc:107] Waiting for socket to connect
0927 14:52:23.465760 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0927 14:52:23.467427 (+ 1667us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:52:23.467973 (+ 546us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:52:23.467981 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:52:23.468085 (+ 104us) client_negotiation.cc:189] Negotiated authn=SASL
0927 14:52:23.468637 (+ 552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:52:23.468650 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:52:23.469424 (+ 774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:52:23.469427 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:52:23.470366 (+ 939us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:52:23.470376 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:52:23.470826 (+ 450us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:52:23.470829 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:52:23.470995 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:52:23.471582 (+ 587us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 14:52:23.471583 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0927 14:52:23.471620 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 14:52:23.471678 (+ 58us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 14:52:23.471938 (+ 260us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 14:52:23.471944 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 14:52:23.471948 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 14:52:23.472036 (+ 88us) client_negotiation.cc:240] Negotiation successful
0927 14:52:23.472234 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":252,"thread_start_us":102,"threads_started":1}
I20260927 14:52:23.474023 10086 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41879
I20260927 14:52:23.474211 10086 heartbeater.cc:464] Registering TS with master...
I20260927 14:52:23.474644 10086 heartbeater.cc:511] Master 127.0.98.62:41879 requested a full tablet report, sending...
I20260927 14:52:23.475509 9636 ts_manager.cc:194] Registered new tserver with Master: 702f62a037f6412888ef10938d60b553 (127.0.98.3:32853)
I20260927 14:52:23.476047 9636 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.3:58841
I20260927 14:52:23.479727 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260927 14:52:23.482345 9675 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 14:52:23.481499 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33084 (local address 127.0.98.62:41879)
0927 14:52:23.481627 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0927 14:52:23.481632 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 14:52:23.481764 (+ 132us) server_negotiation.cc:407] Connection header received
0927 14:52:23.481891 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:52:23.481893 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:52:23.481927 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:52:23.481989 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:52:23.482235 (+ 246us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33084: BlockingRecv error: recv got EOF from 127.0.0.1:33084 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
I20260927 14:52:23.482496 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6936m5/build/debug/bin/kudu with pid 9696
I20260927 14:52:23.487947 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6936m5/build/debug/bin/kudu with pid 9827
I20260927 14:52:23.492904 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6936m5/build/debug/bin/kudu with pid 9958
I20260927 14:52:23.498216 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6936m5/build/debug/bin/kudu with pid 9605
2026-09-27T14:52:23Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (836 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-09-27T14:52:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T14:52:23Z Disabled control of system clock
I20260927 14:52:23.526087 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6936m5/build/debug/bin/kudu
/tmp/dist-test-task6936m5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790520684602251-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790520684602251-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790520684602251-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790520684602251-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41781
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:36975
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:41781
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260927 14:52:23.632360 10097 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 14:52:23.632606 10097 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 14:52:23.632642 10097 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 14:52:23.636479 10097 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 14:52:23.636536 10097 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 14:52:23.636551 10097 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 14:52:23.636566 10097 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 14:52:23.636581 10097 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 14:52:23.641139 10097 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36975
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790520684602251-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790520684602251-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:41781
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41781
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790520684602251-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10097
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790520684602251-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:15:17 UTC on ccycloud.quasar-ledxfe.root.comops.site
build id 4676134
I20260927 14:52:23.642062 10097 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 14:52:23.643313 10097 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 14:52:23.651835 10102 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 14:52:23.651886 10097 server_base.cc:1034] running on GCE node
W20260927 14:52:23.651857 10103 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 14:52:23.651835 10105 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 14:52:23.652652 10097 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 14:52:23.653790 10097 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 14:52:23.655006 10097 hybrid_clock.cc:648] HybridClock initialized: now 1790520743654989 us; error 27 us; skew 500 ppm
I20260927 14:52:23.656946 10097 webserver.cc:460] Webserver started at http://127.0.98.62:40861/ using document root <none> and password file <none>
I20260927 14:52:23.657577 10097 fs_manager.cc:359] Metadata directory not provided
I20260927 14:52:23.657681 10097 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 14:52:23.657925 10097 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 14:52:23.659997 10097 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790520684602251-392-0/minicluster-data/master-0/data/instance:
uuid: "95adaed383264f66a77018fc0f7a1c28"
format_stamp: "Formatted at 2026-09-27 14:52:23 on dist-test-slave-xk1g"
server_key: "a44af1f6e1c7df545b0a7a78f7327557"
server_key_iv: "9b2ede2ea18ef49e43093ae771b73e32"
server_key_version: "encryptionkey@0"
I20260927 14:52:23.660638 10097 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790520684602251-392-0/minicluster-data/master-0/wal/instance:
uuid: "95adaed383264f66a77018fc0f7a1c28"
format_stamp: "Formatted at 2026-09-27 14:52:23 on dist-test-slave-xk1g"
server_key: "a44af1f6e1c7df545b0a7a78f7327557"
server_key_iv: "9b2ede2ea18ef49e43093ae771b73e32"
server_key_version: "encryptionkey@0"
I20260927 14:52:23.664412 10097 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260927 14:52:23.666848 10111 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 14:52:23.668032 10097 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 14:52:23.668169 10097 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790520684602251-392-0/minicluster-data/master-0/data,/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790520684602251-392-0/minicluster-data/master-0/wal
uuid: "95adaed383264f66a77018fc0f7a1c28"
format_stamp: "Formatted at 2026-09-27 14:52:23 on dist-test-slave-xk1g"
server_key: "a44af1f6e1c7df545b0a7a78f7327557"
server_key_iv: "9b2ede2ea18ef49e43093ae771b73e32"
server_key_version: "encryptionkey@0"
I20260927 14:52:23.668288 10097 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790520684602251-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790520684602251-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790520684602251-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 14:52:23.672696 10097 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 14:52:23.673302 10097 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 14:52:23.673477 10097 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 14:52:23.680094 10097 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:41781
I20260927 14:52:23.680101 10163 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:41781 every 8 connection(s)
I20260927 14:52:23.680956 10097 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790520684602251-392-0/minicluster-data/master-0/data/info.pb
I20260927 14:52:23.681330 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6936m5/build/debug/bin/kudu as pid 10097
I20260927 14:52:23.681447 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790520684602251-392-0/minicluster-data/master-0/wal/instance
I20260927 14:52:23.681749 392 external_mini_cluster.cc:1428] Setting key 8e60dbdccbedf57e71205052dd185f7d
I20260927 14:52:23.684130 10164 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 14:52:23.687786 10167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:52:23.682947 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57594 (local address 127.0.98.62:41781)
0927 14:52:23.684000 (+ 1053us) server_negotiation.cc:206] Beginning negotiation
0927 14:52:23.684019 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0927 14:52:23.684057 (+ 38us) server_negotiation.cc:407] Connection header received
0927 14:52:23.684876 (+ 819us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:52:23.684897 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:52:23.685326 (+ 429us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:52:23.685665 (+ 339us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:52:23.686395 (+ 730us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:52:23.686401 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:52:23.686414 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 14:52:23.686426 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:52:23.686486 (+ 60us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 14:52:23.686507 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:52:23.686643 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0927 14:52:23.686706 (+ 63us) server_negotiation.cc:299] Negotiation successful
0927 14:52:23.686939 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":671,"thread_start_us":524,"threads_started":1}
I20260927 14:52:23.690469 10164 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 95adaed383264f66a77018fc0f7a1c28: Bootstrap starting.
I20260927 14:52:23.692332 10164 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 95adaed383264f66a77018fc0f7a1c28: Neither blocks nor log segments found. Creating new log.
I20260927 14:52:23.692945 10164 log.cc:824] T 00000000000000000000000000000000 P 95adaed383264f66a77018fc0f7a1c28: Log is configured to *not* fsync() on all Append() calls
I20260927 14:52:23.694796 10164 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 95adaed383264f66a77018fc0f7a1c28: No bootstrap required, opened a new log
I20260927 14:52:23.698020 10164 raft_consensus.cc:348] T 00000000000000000000000000000000 P 95adaed383264f66a77018fc0f7a1c28 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "95adaed383264f66a77018fc0f7a1c28" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41781 } }
I20260927 14:52:23.698288 10164 raft_consensus.cc:374] T 00000000000000000000000000000000 P 95adaed383264f66a77018fc0f7a1c28 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 14:52:23.698401 10164 raft_consensus.cc:729] T 00000000000000000000000000000000 P 95adaed383264f66a77018fc0f7a1c28 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 95adaed383264f66a77018fc0f7a1c28, State: Initialized, Role: FOLLOWER
I20260927 14:52:23.698812 10164 consensus_queue.cc:260] T 00000000000000000000000000000000 P 95adaed383264f66a77018fc0f7a1c28 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "95adaed383264f66a77018fc0f7a1c28" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41781 } }
I20260927 14:52:23.698925 10164 raft_consensus.cc:388] T 00000000000000000000000000000000 P 95adaed383264f66a77018fc0f7a1c28 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 14:52:23.698962 10164 raft_consensus.cc:482] T 00000000000000000000000000000000 P 95adaed383264f66a77018fc0f7a1c28 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 14:52:23.699036 10164 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 95adaed383264f66a77018fc0f7a1c28 [term 0 FOLLOWER]: Advancing to term 1
I20260927 14:52:23.699784 10164 raft_consensus.cc:504] T 00000000000000000000000000000000 P 95adaed383264f66a77018fc0f7a1c28 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "95adaed383264f66a77018fc0f7a1c28" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41781 } }
I20260927 14:52:23.700048 10164 leader_election.cc:302] T 00000000000000000000000000000000 P 95adaed383264f66a77018fc0f7a1c28 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 95adaed383264f66a77018fc0f7a1c28; no voters:
I20260927 14:52:23.700269 10164 leader_election.cc:288] T 00000000000000000000000000000000 P 95adaed383264f66a77018fc0f7a1c28 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 14:52:23.700397 10169 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 95adaed383264f66a77018fc0f7a1c28 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 14:52:23.700618 10169 raft_consensus.cc:686] T 00000000000000000000000000000000 P 95adaed383264f66a77018fc0f7a1c28 [term 1 LEADER]: Becoming Leader. State: Replica: 95adaed383264f66a77018fc0f7a1c28, State: Running, Role: LEADER
I20260927 14:52:23.700963 10169 consensus_queue.cc:237] T 00000000000000000000000000000000 P 95adaed383264f66a77018fc0f7a1c28 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "95adaed383264f66a77018fc0f7a1c28" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41781 } }
I20260927 14:52:23.701431 10164 sys_catalog.cc:564] T 00000000000000000000000000000000 P 95adaed383264f66a77018fc0f7a1c28 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 14:52:23.702695 10171 sys_catalog.cc:455] T 00000000000000000000000000000000 P 95adaed383264f66a77018fc0f7a1c28 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 95adaed383264f66a77018fc0f7a1c28. Latest consensus state: current_term: 1 leader_uuid: "95adaed383264f66a77018fc0f7a1c28" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "95adaed383264f66a77018fc0f7a1c28" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41781 } } }
I20260927 14:52:23.702818 10171 sys_catalog.cc:458] T 00000000000000000000000000000000 P 95adaed383264f66a77018fc0f7a1c28 [sys.catalog]: This master's current role is: LEADER
I20260927 14:52:23.703197 10178 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 14:52:23.703542 10170 sys_catalog.cc:455] T 00000000000000000000000000000000 P 95adaed383264f66a77018fc0f7a1c28 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "95adaed383264f66a77018fc0f7a1c28" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "95adaed383264f66a77018fc0f7a1c28" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41781 } } }
I20260927 14:52:23.703629 10170 sys_catalog.cc:458] T 00000000000000000000000000000000 P 95adaed383264f66a77018fc0f7a1c28 [sys.catalog]: This master's current role is: LEADER
I20260927 14:52:23.706226 10178 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 14:52:23.711352 10178 catalog_manager.cc:1347] Generated new cluster ID: 526897d97a274c8e99820d7d979cd488
I20260927 14:52:23.711416 10178 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 14:52:23.726028 10178 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 14:52:23.726356 10178 catalog_manager.cc:1504] Loading token signing keys...
I20260927 14:52:23.740772 10178 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 95adaed383264f66a77018fc0f7a1c28: Generated new TSK 0
I20260927 14:52:23.741648 10178 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 14:52:23.749538 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6936m5/build/debug/bin/kudu
/tmp/dist-test-task6936m5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790520684602251-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790520684602251-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790520684602251-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790520684602251-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41781
--builtin_ntp_servers=127.0.98.20:36975
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260927 14:52:23.860378 10188 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 14:52:23.860611 10188 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 14:52:23.860649 10188 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 14:52:23.864641 10188 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 14:52:23.864693 10188 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 14:52:23.864778 10188 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 14:52:23.869369 10188 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36975
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790520684602251-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790520684602251-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790520684602251-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41781
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10188
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790520684602251-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:15:17 UTC on ccycloud.quasar-ledxfe.root.comops.site
build id 4676134
I20260927 14:52:23.870311 10188 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 14:52:23.871639 10188 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 14:52:23.880851 10188 server_base.cc:1034] running on GCE node
W20260927 14:52:23.880844 10194 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 14:52:23.880806 10193 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 14:52:23.880806 10196 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 14:52:23.881372 10188 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 14:52:23.881999 10188 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 14:52:23.883188 10188 hybrid_clock.cc:648] HybridClock initialized: now 1790520743883156 us; error 47 us; skew 500 ppm
I20260927 14:52:23.884985 10188 webserver.cc:460] Webserver started at http://127.0.98.1:33909/ using document root <none> and password file <none>
I20260927 14:52:23.885834 10188 fs_manager.cc:359] Metadata directory not provided
I20260927 14:52:23.885939 10188 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 14:52:23.886205 10188 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 14:52:23.888152 10188 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790520684602251-392-0/minicluster-data/ts-0/data/instance:
uuid: "80568d1a4cb342289bc4b34135f68eb7"
format_stamp: "Formatted at 2026-09-27 14:52:23 on dist-test-slave-xk1g"
server_key: "d871f7ec95af34b2667ff9938f7da499"
server_key_iv: "0bcc67ebc708e36c1008dd324ca7031f"
server_key_version: "encryptionkey@0"
I20260927 14:52:23.888681 10188 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790520684602251-392-0/minicluster-data/ts-0/wal/instance:
uuid: "80568d1a4cb342289bc4b34135f68eb7"
format_stamp: "Formatted at 2026-09-27 14:52:23 on dist-test-slave-xk1g"
server_key: "d871f7ec95af34b2667ff9938f7da499"
server_key_iv: "0bcc67ebc708e36c1008dd324ca7031f"
server_key_version: "encryptionkey@0"
I20260927 14:52:23.893477 10188 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260927 14:52:23.896309 10202 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 14:52:23.897493 10188 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 14:52:23.897614 10188 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790520684602251-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790520684602251-392-0/minicluster-data/ts-0/wal
uuid: "80568d1a4cb342289bc4b34135f68eb7"
format_stamp: "Formatted at 2026-09-27 14:52:23 on dist-test-slave-xk1g"
server_key: "d871f7ec95af34b2667ff9938f7da499"
server_key_iv: "0bcc67ebc708e36c1008dd324ca7031f"
server_key_version: "encryptionkey@0"
I20260927 14:52:23.897732 10188 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790520684602251-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790520684602251-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790520684602251-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 14:52:23.902307 10188 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 14:52:23.902880 10188 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 14:52:23.903055 10188 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 14:52:23.903625 10188 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 14:52:23.904495 10188 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 14:52:23.904564 10188 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 14:52:23.904629 10188 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 14:52:23.904663 10188 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 14:52:23.913283 10188 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:43009
I20260927 14:52:23.913362 10315 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:43009 every 8 connection(s)
I20260927 14:52:23.914208 10188 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790520684602251-392-0/minicluster-data/ts-0/data/info.pb
I20260927 14:52:23.915594 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6936m5/build/debug/bin/kudu as pid 10188
I20260927 14:52:23.915683 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790520684602251-392-0/minicluster-data/ts-0/wal/instance
I20260927 14:52:23.916030 392 external_mini_cluster.cc:1428] Setting key f25bddc6bf851e984c55d3b9a5578eb3
I20260927 14:52:23.917927 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6936m5/build/debug/bin/kudu
/tmp/dist-test-task6936m5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790520684602251-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790520684602251-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790520684602251-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790520684602251-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41781
--builtin_ntp_servers=127.0.98.20:36975
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260927 14:52:23.920885 10167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:52:23.916077 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:54701 (local address 127.0.98.62:41781)
0927 14:52:23.916443 (+ 366us) server_negotiation.cc:206] Beginning negotiation
0927 14:52:23.916447 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 14:52:23.917470 (+ 1023us) server_negotiation.cc:407] Connection header received
0927 14:52:23.918848 (+ 1378us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:52:23.918851 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:52:23.918890 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:52:23.918967 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:52:23.920352 (+ 1385us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:52:23.920358 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:52:23.920364 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 14:52:23.920381 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:52:23.920409 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 14:52:23.920422 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:52:23.920522 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0927 14:52:23.920641 (+ 119us) server_negotiation.cc:299] Negotiation successful
0927 14:52:23.920689 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":280}
I20260927 14:52:23.921663 10318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:52:23.916572 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41781 (local address 127.0.98.1:54701)
0927 14:52:23.917328 (+ 756us) negotiation.cc:107] Waiting for socket to connect
0927 14:52:23.917379 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0927 14:52:23.918642 (+ 1263us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:52:23.919154 (+ 512us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:52:23.919166 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:52:23.919301 (+ 135us) client_negotiation.cc:189] Negotiated authn=SASL
0927 14:52:23.919987 (+ 686us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 14:52:23.919991 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 14:52:23.920052 (+ 61us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 14:52:23.920148 (+ 96us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 14:52:23.920514 (+ 366us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 14:52:23.920521 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 14:52:23.920524 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 14:52:23.920657 (+ 133us) client_negotiation.cc:240] Negotiation successful
0927 14:52:23.920852 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":278,"thread_start_us":131,"threads_started":1}
I20260927 14:52:23.922895 10316 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41781
I20260927 14:52:23.923127 10316 heartbeater.cc:464] Registering TS with master...
I20260927 14:52:23.923554 10316 heartbeater.cc:511] Master 127.0.98.62:41781 requested a full tablet report, sending...
I20260927 14:52:23.925161 10128 ts_manager.cc:194] Registered new tserver with Master: 80568d1a4cb342289bc4b34135f68eb7 (127.0.98.1:43009)
W20260927 14:52:24.026432 10319 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 14:52:24.026707 10319 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 14:52:24.026779 10319 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 14:52:24.030748 10319 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 14:52:24.030825 10319 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 14:52:24.030912 10319 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 14:52:24.035687 10319 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36975
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790520684602251-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790520684602251-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790520684602251-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41781
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10319
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790520684602251-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:15:17 UTC on ccycloud.quasar-ledxfe.root.comops.site
build id 4676134
I20260927 14:52:24.036751 10319 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 14:52:24.038162 10319 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 14:52:24.047475 10325 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 14:52:24.047513 10319 server_base.cc:1034] running on GCE node
W20260927 14:52:24.047487 10327 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 14:52:24.047470 10324 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 14:52:24.048144 10319 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 14:52:24.048826 10319 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 14:52:24.050012 10319 hybrid_clock.cc:648] HybridClock initialized: now 1790520744049983 us; error 40 us; skew 500 ppm
I20260927 14:52:24.051813 10319 webserver.cc:460] Webserver started at http://127.0.98.2:42615/ using document root <none> and password file <none>
I20260927 14:52:24.052397 10319 fs_manager.cc:359] Metadata directory not provided
I20260927 14:52:24.052491 10319 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 14:52:24.052703 10319 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 14:52:24.054764 10319 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790520684602251-392-0/minicluster-data/ts-1/data/instance:
uuid: "8cb0c7de022a488caee1e661ec006cb8"
format_stamp: "Formatted at 2026-09-27 14:52:24 on dist-test-slave-xk1g"
server_key: "dae36c9b88efe4fcbcc28ea16a57d83b"
server_key_iv: "3361029fd6b26f7829b807c79f60b820"
server_key_version: "encryptionkey@0"
I20260927 14:52:24.055325 10319 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790520684602251-392-0/minicluster-data/ts-1/wal/instance:
uuid: "8cb0c7de022a488caee1e661ec006cb8"
format_stamp: "Formatted at 2026-09-27 14:52:24 on dist-test-slave-xk1g"
server_key: "dae36c9b88efe4fcbcc28ea16a57d83b"
server_key_iv: "3361029fd6b26f7829b807c79f60b820"
server_key_version: "encryptionkey@0"
I20260927 14:52:24.058877 10319 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.002s
I20260927 14:52:24.061179 10333 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 14:52:24.062309 10319 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 14:52:24.062441 10319 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790520684602251-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790520684602251-392-0/minicluster-data/ts-1/wal
uuid: "8cb0c7de022a488caee1e661ec006cb8"
format_stamp: "Formatted at 2026-09-27 14:52:24 on dist-test-slave-xk1g"
server_key: "dae36c9b88efe4fcbcc28ea16a57d83b"
server_key_iv: "3361029fd6b26f7829b807c79f60b820"
server_key_version: "encryptionkey@0"
I20260927 14:52:24.062551 10319 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790520684602251-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790520684602251-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790520684602251-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 14:52:24.066984 10319 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 14:52:24.067548 10319 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 14:52:24.067727 10319 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 14:52:24.068246 10319 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 14:52:24.069130 10319 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 14:52:24.069193 10319 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 14:52:24.069262 10319 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 14:52:24.069312 10319 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 14:52:24.077535 10319 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:45263
I20260927 14:52:24.077579 10446 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:45263 every 8 connection(s)
I20260927 14:52:24.078370 10319 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790520684602251-392-0/minicluster-data/ts-1/data/info.pb
I20260927 14:52:24.083938 10167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:52:24.080006 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:48107 (local address 127.0.98.62:41781)
0927 14:52:24.080158 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0927 14:52:24.080164 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 14:52:24.081223 (+ 1059us) server_negotiation.cc:407] Connection header received
0927 14:52:24.082453 (+ 1230us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:52:24.082456 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:52:24.082491 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:52:24.082563 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:52:24.083560 (+ 997us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:52:24.083562 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:52:24.083564 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 14:52:24.083575 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:52:24.083592 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 14:52:24.083600 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:52:24.083672 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0927 14:52:24.083797 (+ 125us) server_negotiation.cc:299] Negotiation successful
0927 14:52:24.083835 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260927 14:52:24.084273 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6936m5/build/debug/bin/kudu as pid 10319
I20260927 14:52:24.084394 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790520684602251-392-0/minicluster-data/ts-1/wal/instance
I20260927 14:52:24.084681 392 external_mini_cluster.cc:1428] Setting key f0c946b1a2c5ced696e8a48b407df211
I20260927 14:52:24.084955 10449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:52:24.080392 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41781 (local address 127.0.98.2:48107)
0927 14:52:24.081024 (+ 632us) negotiation.cc:107] Waiting for socket to connect
0927 14:52:24.081081 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0927 14:52:24.082276 (+ 1195us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:52:24.082715 (+ 439us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:52:24.082738 (+ 23us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:52:24.082823 (+ 85us) client_negotiation.cc:189] Negotiated authn=SASL
0927 14:52:24.083321 (+ 498us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 14:52:24.083322 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0927 14:52:24.083370 (+ 48us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 14:52:24.083433 (+ 63us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 14:52:24.083671 (+ 238us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 14:52:24.083677 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 14:52:24.083680 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 14:52:24.083817 (+ 137us) client_negotiation.cc:240] Negotiation successful
0927 14:52:24.083992 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":236,"thread_start_us":95,"threads_started":1}
I20260927 14:52:24.086050 10447 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41781
I20260927 14:52:24.086284 10447 heartbeater.cc:464] Registering TS with master...
I20260927 14:52:24.086702 10447 heartbeater.cc:511] Master 127.0.98.62:41781 requested a full tablet report, sending...
I20260927 14:52:24.086741 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6936m5/build/debug/bin/kudu
/tmp/dist-test-task6936m5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790520684602251-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790520684602251-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790520684602251-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790520684602251-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41781
--builtin_ntp_servers=127.0.98.20:36975
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260927 14:52:24.087746 10128 ts_manager.cc:194] Registered new tserver with Master: 8cb0c7de022a488caee1e661ec006cb8 (127.0.98.2:45263)
W20260927 14:52:24.193128 10450 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 14:52:24.193436 10450 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 14:52:24.193509 10450 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 14:52:24.197593 10450 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 14:52:24.197672 10450 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 14:52:24.197760 10450 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 14:52:24.203027 10450 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36975
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790520684602251-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790520684602251-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790520684602251-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41781
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10450
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790520684602251-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:15:17 UTC on ccycloud.quasar-ledxfe.root.comops.site
build id 4676134
I20260927 14:52:24.204126 10450 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 14:52:24.205567 10450 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 14:52:24.214941 10455 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 14:52:24.215088 10458 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 14:52:24.215094 10456 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 14:52:24.215142 10450 server_base.cc:1034] running on GCE node
I20260927 14:52:24.215734 10450 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 14:52:24.216403 10450 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 14:52:24.217584 10450 hybrid_clock.cc:648] HybridClock initialized: now 1790520744217573 us; error 30 us; skew 500 ppm
I20260927 14:52:24.219348 10450 webserver.cc:460] Webserver started at http://127.0.98.3:36175/ using document root <none> and password file <none>
I20260927 14:52:24.219919 10450 fs_manager.cc:359] Metadata directory not provided
I20260927 14:52:24.220010 10450 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 14:52:24.220245 10450 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 14:52:24.222363 10450 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790520684602251-392-0/minicluster-data/ts-2/data/instance:
uuid: "27e62e1e229649b390b2ea20e36acbf3"
format_stamp: "Formatted at 2026-09-27 14:52:24 on dist-test-slave-xk1g"
server_key: "a7b96b2a3630bfa1d46cfb299ff7e51e"
server_key_iv: "f12012719bd6abaa0facddd7ce85a30c"
server_key_version: "encryptionkey@0"
I20260927 14:52:24.222916 10450 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790520684602251-392-0/minicluster-data/ts-2/wal/instance:
uuid: "27e62e1e229649b390b2ea20e36acbf3"
format_stamp: "Formatted at 2026-09-27 14:52:24 on dist-test-slave-xk1g"
server_key: "a7b96b2a3630bfa1d46cfb299ff7e51e"
server_key_iv: "f12012719bd6abaa0facddd7ce85a30c"
server_key_version: "encryptionkey@0"
I20260927 14:52:24.226452 10450 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 14:52:24.228642 10464 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 14:52:24.229868 10450 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 14:52:24.230002 10450 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790520684602251-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790520684602251-392-0/minicluster-data/ts-2/wal
uuid: "27e62e1e229649b390b2ea20e36acbf3"
format_stamp: "Formatted at 2026-09-27 14:52:24 on dist-test-slave-xk1g"
server_key: "a7b96b2a3630bfa1d46cfb299ff7e51e"
server_key_iv: "f12012719bd6abaa0facddd7ce85a30c"
server_key_version: "encryptionkey@0"
I20260927 14:52:24.230113 10450 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790520684602251-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790520684602251-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790520684602251-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 14:52:24.234676 10450 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 14:52:24.235284 10450 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 14:52:24.235469 10450 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 14:52:24.236028 10450 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 14:52:24.236933 10450 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 14:52:24.237005 10450 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 14:52:24.237068 10450 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 14:52:24.237125 10450 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 14:52:24.245860 10450 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:42439
I20260927 14:52:24.245932 10577 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:42439 every 8 connection(s)
I20260927 14:52:24.246752 10450 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790520684602251-392-0/minicluster-data/ts-2/data/info.pb
I20260927 14:52:24.252621 10167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:52:24.248543 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:60445 (local address 127.0.98.62:41781)
0927 14:52:24.248683 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0927 14:52:24.248687 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 14:52:24.249821 (+ 1134us) server_negotiation.cc:407] Connection header received
0927 14:52:24.251020 (+ 1199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:52:24.251023 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:52:24.251058 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:52:24.251134 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:52:24.252160 (+ 1026us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:52:24.252163 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:52:24.252165 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 14:52:24.252176 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:52:24.252195 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 14:52:24.252202 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:52:24.252280 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0927 14:52:24.252428 (+ 148us) server_negotiation.cc:299] Negotiation successful
0927 14:52:24.252475 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260927 14:52:24.252630 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6936m5/build/debug/bin/kudu as pid 10450
I20260927 14:52:24.252724 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790520684602251-392-0/minicluster-data/ts-2/wal/instance
I20260927 14:52:24.252995 392 external_mini_cluster.cc:1428] Setting key 8d9341001c1a958bfe46d103b5ddcf34
I20260927 14:52:24.253536 10580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:52:24.248988 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41781 (local address 127.0.98.3:60445)
0927 14:52:24.249677 (+ 689us) negotiation.cc:107] Waiting for socket to connect
0927 14:52:24.249714 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0927 14:52:24.250838 (+ 1124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:52:24.251283 (+ 445us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:52:24.251289 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:52:24.251383 (+ 94us) client_negotiation.cc:189] Negotiated authn=SASL
0927 14:52:24.251889 (+ 506us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 14:52:24.251890 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0927 14:52:24.251929 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 14:52:24.251995 (+ 66us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 14:52:24.252277 (+ 282us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 14:52:24.252283 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 14:52:24.252286 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 14:52:24.252435 (+ 149us) client_negotiation.cc:240] Negotiation successful
0927 14:52:24.252618 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":312,"thread_start_us":122,"threads_started":1}
I20260927 14:52:24.254626 10578 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41781
I20260927 14:52:24.254877 10578 heartbeater.cc:464] Registering TS with master...
I20260927 14:52:24.255308 10578 heartbeater.cc:511] Master 127.0.98.62:41781 requested a full tablet report, sending...
I20260927 14:52:24.256215 10128 ts_manager.cc:194] Registered new tserver with Master: 27e62e1e229649b390b2ea20e36acbf3 (127.0.98.3:42439)
I20260927 14:52:24.267131 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260927 14:52:24.270066 10167 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 14:52:24.269145 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57602 (local address 127.0.98.62:41781)
0927 14:52:24.269291 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0927 14:52:24.269297 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 14:52:24.269444 (+ 147us) server_negotiation.cc:407] Connection header received
0927 14:52:24.269548 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:52:24.269550 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:52:24.269582 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:52:24.269647 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:52:24.269908 (+ 261us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57602: BlockingRecv error: recv got EOF from 127.0.0.1:57602 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
I20260927 14:52:24.270147 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6936m5/build/debug/bin/kudu with pid 10188
I20260927 14:52:24.275768 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6936m5/build/debug/bin/kudu with pid 10319
I20260927 14:52:24.280972 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6936m5/build/debug/bin/kudu with pid 10450
I20260927 14:52:24.286193 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6936m5/build/debug/bin/kudu with pid 10097
2026-09-27T14:52:24Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (787 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790520684602251-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 14:52:24 dist-test-slave-xk1g krb5kdc[10588](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 14:52:24 dist-test-slave-xk1g krb5kdc[10588](info): set up 2 sockets
Sep 27 14:52:24 dist-test-slave-xk1g krb5kdc[10588](info): commencing operation
krb5kdc: starting...
W20260927 14:52:25.325937 392 mini_kdc.cc:121] Time spent starting KDC: real 1.019s user 0.005s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790520684602251-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790520684602251-392-0/krb5kdc/test-user.keytab.
Sep 27 14:52:25 dist-test-slave-xk1g krb5kdc[10588](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790520745, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T14:52:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T14:52:25Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790520684602251-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790520684602251-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 14:52:25.443490 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6936m5/build/debug/bin/kudu
/tmp/dist-test-task6936m5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790520684602251-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790520684602251-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790520684602251-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790520684602251-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39611
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790520684602251-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35859
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:39611
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790520684602251-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790520684602251-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790520684602251-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 14:52:25.548942 10601 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 14:52:25.549259 10601 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 14:52:25.549345 10601 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 14:52:25.553360 10601 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 14:52:25.553437 10601 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 14:52:25.553474 10601 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 14:52:25.553530 10601 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 14:52:25.553565 10601 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 14:52:25.558774 10601 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35859
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790520684602251-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790520684602251-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:39611
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790520684602251-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39611
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790520684602251-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10601
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790520684602251-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:15:17 UTC on ccycloud.quasar-ledxfe.root.comops.site
build id 4676134
I20260927 14:52:25.559785 10601 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 14:52:25.560817 10601 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 14:52:25.569154 10609 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 14:52:25.569175 10607 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 14:52:25.569175 10606 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 14:52:25.569305 10601 server_base.cc:1034] running on GCE node
I20260927 14:52:25.569979 10601 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 14:52:25.571024 10601 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 14:52:25.572209 10601 hybrid_clock.cc:648] HybridClock initialized: now 1790520745572185 us; error 41 us; skew 500 ppm
Sep 27 14:52:25 dist-test-slave-xk1g krb5kdc[10588](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790520745, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 14:52:25.574791 10601 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 14:52:25.575635 10601 webserver.cc:460] Webserver started at http://127.0.98.62:43687/ using document root <none> and password file <none>
I20260927 14:52:25.576185 10601 fs_manager.cc:359] Metadata directory not provided
I20260927 14:52:25.576270 10601 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 14:52:25.576490 10601 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 14:52:25.578454 10601 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790520684602251-392-0/minicluster-data/master-0/data/instance:
uuid: "cd0e61e12fc94075901d6599bb52df98"
format_stamp: "Formatted at 2026-09-27 14:52:25 on dist-test-slave-xk1g"
server_key: "de0a3f2e572d8ca1d9b820a1e8a04dce"
server_key_iv: "ef51cc8d9b3e869130666e4a79447f3b"
server_key_version: "encryptionkey@0"
I20260927 14:52:25.578991 10601 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790520684602251-392-0/minicluster-data/master-0/wal/instance:
uuid: "cd0e61e12fc94075901d6599bb52df98"
format_stamp: "Formatted at 2026-09-27 14:52:25 on dist-test-slave-xk1g"
server_key: "de0a3f2e572d8ca1d9b820a1e8a04dce"
server_key_iv: "ef51cc8d9b3e869130666e4a79447f3b"
server_key_version: "encryptionkey@0"
I20260927 14:52:25.582547 10601 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 14:52:25.584842 10616 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 14:52:25.585991 10601 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 14:52:25.586122 10601 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790520684602251-392-0/minicluster-data/master-0/data,/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790520684602251-392-0/minicluster-data/master-0/wal
uuid: "cd0e61e12fc94075901d6599bb52df98"
format_stamp: "Formatted at 2026-09-27 14:52:25 on dist-test-slave-xk1g"
server_key: "de0a3f2e572d8ca1d9b820a1e8a04dce"
server_key_iv: "ef51cc8d9b3e869130666e4a79447f3b"
server_key_version: "encryptionkey@0"
I20260927 14:52:25.586222 10601 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790520684602251-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790520684602251-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790520684602251-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 14:52:25.605142 10601 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 14:52:25.607928 10601 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 14:52:25.608124 10601 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 14:52:25.615051 10601 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:39611
I20260927 14:52:25.615324 10668 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:39611 every 8 connection(s)
I20260927 14:52:25.616014 10601 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790520684602251-392-0/minicluster-data/master-0/data/info.pb
I20260927 14:52:25.618814 10669 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 14:52:25.618945 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6936m5/build/debug/bin/kudu as pid 10601
I20260927 14:52:25.619052 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790520684602251-392-0/minicluster-data/master-0/wal/instance
I20260927 14:52:25.619345 392 external_mini_cluster.cc:1428] Setting key f42015047d07a68bf3920a8bc28a67e4
I20260927 14:52:25.625911 10669 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cd0e61e12fc94075901d6599bb52df98: Bootstrap starting.
I20260927 14:52:25.628324 10669 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cd0e61e12fc94075901d6599bb52df98: Neither blocks nor log segments found. Creating new log.
I20260927 14:52:25.629221 10669 log.cc:824] T 00000000000000000000000000000000 P cd0e61e12fc94075901d6599bb52df98: Log is configured to *not* fsync() on all Append() calls
Sep 27 14:52:25 dist-test-slave-xk1g krb5kdc[10588](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790520745, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 14:52:25.631064 10669 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cd0e61e12fc94075901d6599bb52df98: No bootstrap required, opened a new log
I20260927 14:52:25.634557 10672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:52:25.620729 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55240 (local address 127.0.98.62:39611)
0927 14:52:25.621201 (+ 472us) server_negotiation.cc:206] Beginning negotiation
0927 14:52:25.621215 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 14:52:25.621249 (+ 34us) server_negotiation.cc:407] Connection header received
0927 14:52:25.622009 (+ 760us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:52:25.622027 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:52:25.622498 (+ 471us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:52:25.622891 (+ 393us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:52:25.624047 (+ 1156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:52:25.625547 (+ 1500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:52:25.626433 (+ 886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:52:25.627105 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:52:25.627281 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:52:25.629999 (+ 2718us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:52:25.630004 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:52:25.630017 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 14:52:25.630041 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:52:25.631761 (+ 1720us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:52:25.632248 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:52:25.632252 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:52:25.632257 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:52:25.632361 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:52:25.632644 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:52:25.632646 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:52:25.632647 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:52:25.633020 (+ 373us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:52:25.633240 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0927 14:52:25.633535 (+ 295us) server_negotiation.cc:299] Negotiation successful
0927 14:52:25.633833 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":332,"thread_start_us":135,"threads_started":1}
I20260927 14:52:25.635107 10669 raft_consensus.cc:348] T 00000000000000000000000000000000 P cd0e61e12fc94075901d6599bb52df98 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd0e61e12fc94075901d6599bb52df98" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39611 } }
I20260927 14:52:25.635355 10669 raft_consensus.cc:374] T 00000000000000000000000000000000 P cd0e61e12fc94075901d6599bb52df98 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 14:52:25.635434 10669 raft_consensus.cc:729] T 00000000000000000000000000000000 P cd0e61e12fc94075901d6599bb52df98 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd0e61e12fc94075901d6599bb52df98, State: Initialized, Role: FOLLOWER
I20260927 14:52:25.635870 10669 consensus_queue.cc:260] T 00000000000000000000000000000000 P cd0e61e12fc94075901d6599bb52df98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd0e61e12fc94075901d6599bb52df98" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39611 } }
I20260927 14:52:25.636020 10669 raft_consensus.cc:388] T 00000000000000000000000000000000 P cd0e61e12fc94075901d6599bb52df98 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 14:52:25.636104 10669 raft_consensus.cc:482] T 00000000000000000000000000000000 P cd0e61e12fc94075901d6599bb52df98 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 14:52:25.636219 10669 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cd0e61e12fc94075901d6599bb52df98 [term 0 FOLLOWER]: Advancing to term 1
I20260927 14:52:25.637321 10669 raft_consensus.cc:504] T 00000000000000000000000000000000 P cd0e61e12fc94075901d6599bb52df98 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd0e61e12fc94075901d6599bb52df98" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39611 } }
I20260927 14:52:25.637655 10669 leader_election.cc:302] T 00000000000000000000000000000000 P cd0e61e12fc94075901d6599bb52df98 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cd0e61e12fc94075901d6599bb52df98; no voters:
I20260927 14:52:25.637955 10669 leader_election.cc:288] T 00000000000000000000000000000000 P cd0e61e12fc94075901d6599bb52df98 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 14:52:25.638079 10674 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cd0e61e12fc94075901d6599bb52df98 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 14:52:25.638327 10674 raft_consensus.cc:686] T 00000000000000000000000000000000 P cd0e61e12fc94075901d6599bb52df98 [term 1 LEADER]: Becoming Leader. State: Replica: cd0e61e12fc94075901d6599bb52df98, State: Running, Role: LEADER
I20260927 14:52:25.638614 10674 consensus_queue.cc:237] T 00000000000000000000000000000000 P cd0e61e12fc94075901d6599bb52df98 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd0e61e12fc94075901d6599bb52df98" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39611 } }
I20260927 14:52:25.639163 10669 sys_catalog.cc:564] T 00000000000000000000000000000000 P cd0e61e12fc94075901d6599bb52df98 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 14:52:25.640317 10675 sys_catalog.cc:455] T 00000000000000000000000000000000 P cd0e61e12fc94075901d6599bb52df98 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cd0e61e12fc94075901d6599bb52df98" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd0e61e12fc94075901d6599bb52df98" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39611 } } }
I20260927 14:52:25.640314 10676 sys_catalog.cc:455] T 00000000000000000000000000000000 P cd0e61e12fc94075901d6599bb52df98 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cd0e61e12fc94075901d6599bb52df98. Latest consensus state: current_term: 1 leader_uuid: "cd0e61e12fc94075901d6599bb52df98" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd0e61e12fc94075901d6599bb52df98" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39611 } } }
I20260927 14:52:25.640465 10676 sys_catalog.cc:458] T 00000000000000000000000000000000 P cd0e61e12fc94075901d6599bb52df98 [sys.catalog]: This master's current role is: LEADER
I20260927 14:52:25.640465 10675 sys_catalog.cc:458] T 00000000000000000000000000000000 P cd0e61e12fc94075901d6599bb52df98 [sys.catalog]: This master's current role is: LEADER
I20260927 14:52:25.640854 10683 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 14:52:25.643498 10683 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 14:52:25.648849 10683 catalog_manager.cc:1347] Generated new cluster ID: 6edc0abe766546ddb997783c2ac2867e
I20260927 14:52:25.648917 10683 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 14:52:25.660317 10683 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 14:52:25.661388 10683 catalog_manager.cc:1504] Loading token signing keys...
I20260927 14:52:25.666126 10683 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cd0e61e12fc94075901d6599bb52df98: Generated new TSK 0
I20260927 14:52:25.666831 10683 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790520684602251-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790520684602251-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260927 14:52:25.694131 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6936m5/build/debug/bin/kudu
/tmp/dist-test-task6936m5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790520684602251-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790520684602251-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790520684602251-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790520684602251-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39611
--keytab_file=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790520684602251-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35859
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790520684602251-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790520684602251-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790520684602251-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 14:52:25.802423 10695 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 14:52:25.802745 10695 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 14:52:25.802834 10695 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 14:52:25.806993 10695 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 14:52:25.807080 10695 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 14:52:25.807174 10695 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 14:52:25.812260 10695 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35859
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790520684602251-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790520684602251-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790520684602251-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790520684602251-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39611
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10695
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790520684602251-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:15:17 UTC on ccycloud.quasar-ledxfe.root.comops.site
build id 4676134
I20260927 14:52:25.813364 10695 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 14:52:25.814343 10695 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 14:52:25.824370 10700 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 14:52:25.824435 10703 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 14:52:25.824363 10701 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 14:52:25.824601 10695 server_base.cc:1034] running on GCE node
I20260927 14:52:25.825100 10695 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 14:52:25.825757 10695 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 14:52:25.826926 10695 hybrid_clock.cc:648] HybridClock initialized: now 1790520745826914 us; error 27 us; skew 500 ppm
Sep 27 14:52:25 dist-test-slave-xk1g krb5kdc[10588](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790520745, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 14:52:25.829677 10695 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 14:52:25.830524 10695 webserver.cc:460] Webserver started at http://127.0.98.1:37889/ using document root <none> and password file <none>
I20260927 14:52:25.831023 10695 fs_manager.cc:359] Metadata directory not provided
I20260927 14:52:25.831079 10695 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 14:52:25.831244 10695 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 14:52:25.833277 10695 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790520684602251-392-0/minicluster-data/ts-0/data/instance:
uuid: "ce29d918e07c44a9befb799462f534d4"
format_stamp: "Formatted at 2026-09-27 14:52:25 on dist-test-slave-xk1g"
server_key: "5e71ecb49f4adf2f0a9a9ba38812d936"
server_key_iv: "b7c9799a78ba810e7156aca48dfea507"
server_key_version: "encryptionkey@0"
I20260927 14:52:25.833779 10695 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790520684602251-392-0/minicluster-data/ts-0/wal/instance:
uuid: "ce29d918e07c44a9befb799462f534d4"
format_stamp: "Formatted at 2026-09-27 14:52:25 on dist-test-slave-xk1g"
server_key: "5e71ecb49f4adf2f0a9a9ba38812d936"
server_key_iv: "b7c9799a78ba810e7156aca48dfea507"
server_key_version: "encryptionkey@0"
I20260927 14:52:25.837244 10695 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260927 14:52:25.839709 10710 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 14:52:25.841032 10695 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260927 14:52:25.841223 10695 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790520684602251-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790520684602251-392-0/minicluster-data/ts-0/wal
uuid: "ce29d918e07c44a9befb799462f534d4"
format_stamp: "Formatted at 2026-09-27 14:52:25 on dist-test-slave-xk1g"
server_key: "5e71ecb49f4adf2f0a9a9ba38812d936"
server_key_iv: "b7c9799a78ba810e7156aca48dfea507"
server_key_version: "encryptionkey@0"
I20260927 14:52:25.841341 10695 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790520684602251-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790520684602251-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790520684602251-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 14:52:25.862733 10695 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 14:52:25.865537 10695 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 14:52:25.865756 10695 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 14:52:25.866330 10695 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 14:52:25.867137 10695 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 14:52:25.867208 10695 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 14:52:25.867282 10695 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 14:52:25.867324 10695 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 14:52:25.875798 10695 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:43159
I20260927 14:52:25.875818 10823 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:43159 every 8 connection(s)
I20260927 14:52:25.876672 10695 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790520684602251-392-0/minicluster-data/ts-0/data/info.pb
I20260927 14:52:25.880707 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6936m5/build/debug/bin/kudu as pid 10695
I20260927 14:52:25.880822 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790520684602251-392-0/minicluster-data/ts-0/wal/instance
I20260927 14:52:25.881160 392 external_mini_cluster.cc:1428] Setting key 745bc69eb560f50520b0b189a238f31c
Sep 27 14:52:25 dist-test-slave-xk1g krb5kdc[10588](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790520745, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260927 14:52:25.893267 10672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:52:25.878490 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:44241 (local address 127.0.98.62:39611)
0927 14:52:25.878616 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0927 14:52:25.878620 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 14:52:25.879552 (+ 932us) server_negotiation.cc:407] Connection header received
0927 14:52:25.880407 (+ 855us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:52:25.880410 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:52:25.880455 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:52:25.880526 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:52:25.882084 (+ 1558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:52:25.882821 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:52:25.884651 (+ 1830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:52:25.884970 (+ 319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:52:25.885096 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:52:25.888689 (+ 3593us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:52:25.888697 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:52:25.888702 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 14:52:25.888741 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:52:25.890858 (+ 2117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:52:25.891513 (+ 655us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:52:25.891517 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:52:25.891519 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:52:25.891604 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:52:25.891957 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:52:25.891961 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:52:25.891962 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:52:25.892190 (+ 228us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:52:25.892288 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0927 14:52:25.892891 (+ 603us) server_negotiation.cc:299] Negotiation successful
0927 14:52:25.892992 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":19,"server-negotiator.queue_time_us":45}
I20260927 14:52:25.894106 10826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:52:25.878808 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39611 (local address 127.0.98.1:44241)
0927 14:52:25.879425 (+ 617us) negotiation.cc:107] Waiting for socket to connect
0927 14:52:25.879466 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0927 14:52:25.880220 (+ 754us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:52:25.880722 (+ 502us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:52:25.880734 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:52:25.881278 (+ 544us) client_negotiation.cc:189] Negotiated authn=SASL
0927 14:52:25.881940 (+ 662us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:52:25.881956 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:52:25.882969 (+ 1013us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:52:25.882974 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:52:25.884511 (+ 1537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:52:25.884528 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:52:25.885303 (+ 775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:52:25.885310 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:52:25.885555 (+ 245us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:52:25.886256 (+ 701us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 14:52:25.886292 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0927 14:52:25.888391 (+ 2099us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 14:52:25.890998 (+ 2607us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 14:52:25.891004 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 14:52:25.891021 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0927 14:52:25.891392 (+ 371us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 14:52:25.891709 (+ 317us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 14:52:25.891712 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 14:52:25.891714 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 14:52:25.891854 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 14:52:25.892320 (+ 466us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 14:52:25.892329 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 14:52:25.892657 (+ 328us) client_negotiation.cc:769] Sending connection context
0927 14:52:25.892828 (+ 171us) client_negotiation.cc:240] Negotiation successful
0927 14:52:25.893239 (+ 411us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":441,"thread_start_us":105,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260927 14:52:25.895423 10824 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39611
I20260927 14:52:25.895675 10824 heartbeater.cc:464] Registering TS with master...
I20260927 14:52:25.896262 10824 heartbeater.cc:511] Master 127.0.98.62:39611 requested a full tablet report, sending...
I20260927 14:52:25.897848 10633 ts_manager.cc:194] Registered new tserver with Master: ce29d918e07c44a9befb799462f534d4 (127.0.98.1:43159)
I20260927 14:52:25.899487 10633 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:44241
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790520684602251-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790520684602251-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260927 14:52:25.910403 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6936m5/build/debug/bin/kudu
/tmp/dist-test-task6936m5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790520684602251-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790520684602251-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790520684602251-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790520684602251-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39611
--keytab_file=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790520684602251-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35859
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790520684602251-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790520684602251-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790520684602251-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 14:52:26.021347 10829 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 14:52:26.021593 10829 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 14:52:26.021636 10829 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 14:52:26.025701 10829 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 14:52:26.025758 10829 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 14:52:26.025816 10829 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 14:52:26.031010 10829 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35859
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790520684602251-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790520684602251-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790520684602251-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790520684602251-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39611
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10829
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790520684602251-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:15:17 UTC on ccycloud.quasar-ledxfe.root.comops.site
build id 4676134
I20260927 14:52:26.032037 10829 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 14:52:26.032997 10829 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 14:52:26.042804 10834 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 14:52:26.042791 10837 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 14:52:26.042791 10835 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 14:52:26.042891 10829 server_base.cc:1034] running on GCE node
I20260927 14:52:26.043512 10829 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 14:52:26.044194 10829 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 14:52:26.045387 10829 hybrid_clock.cc:648] HybridClock initialized: now 1790520746045380 us; error 33 us; skew 500 ppm
Sep 27 14:52:26 dist-test-slave-xk1g krb5kdc[10588](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790520746, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 14:52:26.048363 10829 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 14:52:26.049381 10829 webserver.cc:460] Webserver started at http://127.0.98.2:42695/ using document root <none> and password file <none>
I20260927 14:52:26.049921 10829 fs_manager.cc:359] Metadata directory not provided
I20260927 14:52:26.049979 10829 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 14:52:26.050141 10829 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 14:52:26.052109 10829 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790520684602251-392-0/minicluster-data/ts-1/data/instance:
uuid: "ba41bd7f6425453da6cf1e9a87425c7b"
format_stamp: "Formatted at 2026-09-27 14:52:26 on dist-test-slave-xk1g"
server_key: "8c415d3a2ddb5818681293e36808baef"
server_key_iv: "2ef7b0f7872cd852b7e0e541ba953d36"
server_key_version: "encryptionkey@0"
I20260927 14:52:26.052600 10829 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790520684602251-392-0/minicluster-data/ts-1/wal/instance:
uuid: "ba41bd7f6425453da6cf1e9a87425c7b"
format_stamp: "Formatted at 2026-09-27 14:52:26 on dist-test-slave-xk1g"
server_key: "8c415d3a2ddb5818681293e36808baef"
server_key_iv: "2ef7b0f7872cd852b7e0e541ba953d36"
server_key_version: "encryptionkey@0"
I20260927 14:52:26.056435 10829 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260927 14:52:26.058947 10844 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 14:52:26.060256 10829 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 14:52:26.060391 10829 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790520684602251-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790520684602251-392-0/minicluster-data/ts-1/wal
uuid: "ba41bd7f6425453da6cf1e9a87425c7b"
format_stamp: "Formatted at 2026-09-27 14:52:26 on dist-test-slave-xk1g"
server_key: "8c415d3a2ddb5818681293e36808baef"
server_key_iv: "2ef7b0f7872cd852b7e0e541ba953d36"
server_key_version: "encryptionkey@0"
I20260927 14:52:26.060498 10829 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790520684602251-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790520684602251-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790520684602251-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 14:52:26.082087 10829 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 14:52:26.085485 10829 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 14:52:26.085690 10829 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 14:52:26.086301 10829 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 14:52:26.087158 10829 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 14:52:26.087205 10829 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 14:52:26.087245 10829 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 14:52:26.087302 10829 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 14:52:26.096397 10829 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:35343
I20260927 14:52:26.096414 10957 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:35343 every 8 connection(s)
I20260927 14:52:26.097391 10829 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790520684602251-392-0/minicluster-data/ts-1/data/info.pb
Sep 27 14:52:26 dist-test-slave-xk1g krb5kdc[10588](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790520746, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 14:52:26.106824 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6936m5/build/debug/bin/kudu as pid 10829
I20260927 14:52:26.106945 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790520684602251-392-0/minicluster-data/ts-1/wal/instance
I20260927 14:52:26.107326 392 external_mini_cluster.cc:1428] Setting key a66b771007f172324238b9c9422290c5
I20260927 14:52:26.111123 10672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:52:26.099168 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:42827 (local address 127.0.98.62:39611)
0927 14:52:26.099311 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0927 14:52:26.099314 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 14:52:26.100347 (+ 1033us) server_negotiation.cc:407] Connection header received
0927 14:52:26.101197 (+ 850us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:52:26.101204 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:52:26.101286 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:52:26.101407 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:52:26.102523 (+ 1116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:52:26.103082 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:52:26.104292 (+ 1210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:52:26.104532 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:52:26.104622 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:52:26.107099 (+ 2477us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:52:26.107103 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:52:26.107106 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 14:52:26.107137 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:52:26.108967 (+ 1830us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:52:26.109520 (+ 553us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:52:26.109525 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:52:26.109527 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:52:26.109627 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:52:26.109958 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:52:26.109965 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:52:26.109970 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:52:26.110233 (+ 263us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:52:26.110344 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0927 14:52:26.110827 (+ 483us) server_negotiation.cc:299] Negotiation successful
0927 14:52:26.110973 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260927 14:52:26.112112 10960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:52:26.099557 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39611 (local address 127.0.98.2:42827)
0927 14:52:26.100185 (+ 628us) negotiation.cc:107] Waiting for socket to connect
0927 14:52:26.100226 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0927 14:52:26.100973 (+ 747us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:52:26.101574 (+ 601us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:52:26.101581 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:52:26.101937 (+ 356us) client_negotiation.cc:189] Negotiated authn=SASL
0927 14:52:26.102391 (+ 454us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:52:26.102403 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:52:26.103189 (+ 786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:52:26.103191 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:52:26.104185 (+ 994us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:52:26.104194 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:52:26.104659 (+ 465us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:52:26.104662 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:52:26.104795 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:52:26.105406 (+ 611us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 14:52:26.105422 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 14:52:26.106900 (+ 1478us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 14:52:26.109136 (+ 2236us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 14:52:26.109142 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 14:52:26.109155 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0927 14:52:26.109417 (+ 262us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 14:52:26.109725 (+ 308us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 14:52:26.109727 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 14:52:26.109728 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 14:52:26.109853 (+ 125us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 14:52:26.110347 (+ 494us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 14:52:26.110352 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 14:52:26.110562 (+ 210us) client_negotiation.cc:769] Sending connection context
0927 14:52:26.110707 (+ 145us) client_negotiation.cc:240] Negotiation successful
0927 14:52:26.111056 (+ 349us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":452,"thread_start_us":107,"threads_started":1}
I20260927 14:52:26.113444 10958 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39611
I20260927 14:52:26.113705 10958 heartbeater.cc:464] Registering TS with master...
I20260927 14:52:26.114243 10958 heartbeater.cc:511] Master 127.0.98.62:39611 requested a full tablet report, sending...
I20260927 14:52:26.115325 10633 ts_manager.cc:194] Registered new tserver with Master: ba41bd7f6425453da6cf1e9a87425c7b (127.0.98.2:35343)
I20260927 14:52:26.116114 10633 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:42827
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790520684602251-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790520684602251-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260927 14:52:26.134123 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6936m5/build/debug/bin/kudu
/tmp/dist-test-task6936m5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790520684602251-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790520684602251-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790520684602251-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790520684602251-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39611
--keytab_file=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790520684602251-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35859
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790520684602251-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790520684602251-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790520684602251-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 14:52:26.241871 10963 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 14:52:26.242127 10963 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 14:52:26.242206 10963 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 14:52:26.246347 10963 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 14:52:26.246404 10963 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 14:52:26.246502 10963 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 14:52:26.251901 10963 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35859
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790520684602251-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790520684602251-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790520684602251-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790520684602251-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39611
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10963
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790520684602251-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:15:17 UTC on ccycloud.quasar-ledxfe.root.comops.site
build id 4676134
I20260927 14:52:26.252964 10963 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 14:52:26.253975 10963 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 14:52:26.263371 10968 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 14:52:26.263420 10971 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 14:52:26.263585 10963 server_base.cc:1034] running on GCE node
W20260927 14:52:26.263373 10969 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 14:52:26.264086 10963 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 14:52:26.264698 10963 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 14:52:26.265856 10963 hybrid_clock.cc:648] HybridClock initialized: now 1790520746265841 us; error 28 us; skew 500 ppm
Sep 27 14:52:26 dist-test-slave-xk1g krb5kdc[10588](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790520746, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 14:52:26.268499 10963 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 14:52:26.269425 10963 webserver.cc:460] Webserver started at http://127.0.98.3:39071/ using document root <none> and password file <none>
I20260927 14:52:26.269963 10963 fs_manager.cc:359] Metadata directory not provided
I20260927 14:52:26.270049 10963 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 14:52:26.270270 10963 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 14:52:26.272192 10963 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790520684602251-392-0/minicluster-data/ts-2/data/instance:
uuid: "5aa838a7144d4b1caa83eab5eab0cdb9"
format_stamp: "Formatted at 2026-09-27 14:52:26 on dist-test-slave-xk1g"
server_key: "225986e91402dabc16d618cd15f9092e"
server_key_iv: "6783fbdfd98c302da29bc188ab10c90d"
server_key_version: "encryptionkey@0"
I20260927 14:52:26.272740 10963 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790520684602251-392-0/minicluster-data/ts-2/wal/instance:
uuid: "5aa838a7144d4b1caa83eab5eab0cdb9"
format_stamp: "Formatted at 2026-09-27 14:52:26 on dist-test-slave-xk1g"
server_key: "225986e91402dabc16d618cd15f9092e"
server_key_iv: "6783fbdfd98c302da29bc188ab10c90d"
server_key_version: "encryptionkey@0"
I20260927 14:52:26.276233 10963 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260927 14:52:26.278636 10978 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 14:52:26.280085 10963 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 14:52:26.280210 10963 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790520684602251-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790520684602251-392-0/minicluster-data/ts-2/wal
uuid: "5aa838a7144d4b1caa83eab5eab0cdb9"
format_stamp: "Formatted at 2026-09-27 14:52:26 on dist-test-slave-xk1g"
server_key: "225986e91402dabc16d618cd15f9092e"
server_key_iv: "6783fbdfd98c302da29bc188ab10c90d"
server_key_version: "encryptionkey@0"
I20260927 14:52:26.280323 10963 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790520684602251-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790520684602251-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790520684602251-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 14:52:26.294368 10963 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 14:52:26.297605 10963 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 14:52:26.297816 10963 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 14:52:26.298388 10963 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 14:52:26.299188 10963 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 14:52:26.299232 10963 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 14:52:26.299301 10963 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 14:52:26.299345 10963 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 14:52:26.307930 10963 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:42533
I20260927 14:52:26.308005 11091 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:42533 every 8 connection(s)
I20260927 14:52:26.308795 10963 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790520684602251-392-0/minicluster-data/ts-2/data/info.pb
I20260927 14:52:26.309804 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6936m5/build/debug/bin/kudu as pid 10963
I20260927 14:52:26.309921 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790520684602251-392-0/minicluster-data/ts-2/wal/instance
I20260927 14:52:26.310185 392 external_mini_cluster.cc:1428] Setting key 0873acc33e28f0963cfc32e73fd32304
Sep 27 14:52:26 dist-test-slave-xk1g krb5kdc[10588](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790520746, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 14:52:26.321616 10672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:52:26.310671 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:43325 (local address 127.0.98.62:39611)
0927 14:52:26.310827 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0927 14:52:26.310832 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 14:52:26.312059 (+ 1227us) server_negotiation.cc:407] Connection header received
0927 14:52:26.312945 (+ 886us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:52:26.312947 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:52:26.313002 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:52:26.313116 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:52:26.314197 (+ 1081us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:52:26.314705 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:52:26.315753 (+ 1048us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:52:26.315953 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:52:26.316025 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:52:26.318360 (+ 2335us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:52:26.318363 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:52:26.318365 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 14:52:26.318388 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:52:26.319568 (+ 1180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:52:26.320077 (+ 509us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:52:26.320082 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:52:26.320083 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:52:26.320162 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:52:26.320497 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:52:26.320501 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:52:26.320503 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:52:26.320678 (+ 175us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:52:26.320781 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0927 14:52:26.321331 (+ 550us) server_negotiation.cc:299] Negotiation successful
0927 14:52:26.321449 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260927 14:52:26.322371 11094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:52:26.311176 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39611 (local address 127.0.98.3:43325)
0927 14:52:26.311898 (+ 722us) negotiation.cc:107] Waiting for socket to connect
0927 14:52:26.311952 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0927 14:52:26.312768 (+ 816us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:52:26.313265 (+ 497us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:52:26.313271 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:52:26.313620 (+ 349us) client_negotiation.cc:189] Negotiated authn=SASL
0927 14:52:26.314081 (+ 461us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:52:26.314093 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:52:26.314808 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:52:26.314810 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:52:26.315651 (+ 841us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:52:26.315658 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:52:26.316077 (+ 419us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:52:26.316080 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:52:26.316206 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:52:26.316806 (+ 600us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 14:52:26.316821 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 14:52:26.318192 (+ 1371us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 14:52:26.319687 (+ 1495us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 14:52:26.319693 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 14:52:26.319707 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0927 14:52:26.319972 (+ 265us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 14:52:26.320262 (+ 290us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 14:52:26.320267 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 14:52:26.320269 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 14:52:26.320402 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 14:52:26.320788 (+ 386us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 14:52:26.320797 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 14:52:26.321028 (+ 231us) client_negotiation.cc:769] Sending connection context
0927 14:52:26.321279 (+ 251us) client_negotiation.cc:240] Negotiation successful
0927 14:52:26.321582 (+ 303us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":495,"thread_start_us":110,"threads_started":1}
I20260927 14:52:26.323525 11092 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39611
I20260927 14:52:26.323776 11092 heartbeater.cc:464] Registering TS with master...
I20260927 14:52:26.324258 11092 heartbeater.cc:511] Master 127.0.98.62:39611 requested a full tablet report, sending...
I20260927 14:52:26.325268 10632 ts_manager.cc:194] Registered new tserver with Master: 5aa838a7144d4b1caa83eab5eab0cdb9 (127.0.98.3:42533)
I20260927 14:52:26.325829 10632 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:43325
I20260927 14:52:26.335959 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 14:52:26.345700 10672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:52:26.338090 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55244 (local address 127.0.98.62:39611)
0927 14:52:26.338229 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0927 14:52:26.338233 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 14:52:26.338414 (+ 181us) server_negotiation.cc:407] Connection header received
0927 14:52:26.338528 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:52:26.338530 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:52:26.338569 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:52:26.338680 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:52:26.339534 (+ 854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:52:26.340191 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:52:26.341096 (+ 905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:52:26.341326 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:52:26.341414 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:52:26.342810 (+ 1396us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:52:26.342812 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:52:26.342814 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 14:52:26.342834 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:52:26.344033 (+ 1199us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:52:26.344434 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:52:26.344437 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:52:26.344441 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:52:26.344526 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:52:26.344870 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:52:26.344873 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:52:26.344875 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:52:26.345169 (+ 294us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:52:26.345275 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0927 14:52:26.345450 (+ 175us) server_negotiation.cc:299] Negotiation successful
0927 14:52:26.345573 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260927 14:52:26.348651 10632 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55244:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 14:52:26.351671 10632 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 14:52:26.366636 11103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:52:26.360199 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:35343 (local address 127.0.0.1:33238)
0927 14:52:26.361589 (+ 1390us) negotiation.cc:107] Waiting for socket to connect
0927 14:52:26.361601 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 14:52:26.361711 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:52:26.362491 (+ 780us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:52:26.362494 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:52:26.362510 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 14:52:26.362623 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:52:26.362629 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:52:26.363819 (+ 1190us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:52:26.363822 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:52:26.365189 (+ 1367us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:52:26.365199 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:52:26.366206 (+ 1007us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:52:26.366210 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:52:26.366306 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:52:26.366340 (+ 34us) client_negotiation.cc:769] Sending connection context
0927 14:52:26.366413 (+ 73us) client_negotiation.cc:240] Negotiation successful
0927 14:52:26.366498 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1322,"mutex_wait_us":45,"thread_start_us":120,"threads_started":1}
I20260927 14:52:26.366644 11106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:52:26.360195 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48882 (local address 127.0.98.3:42533)
0927 14:52:26.361590 (+ 1395us) server_negotiation.cc:206] Beginning negotiation
0927 14:52:26.361604 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 14:52:26.361631 (+ 27us) server_negotiation.cc:407] Connection header received
0927 14:52:26.361718 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:52:26.361723 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:52:26.361963 (+ 240us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:52:26.362120 (+ 157us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 14:52:26.362475 (+ 355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:52:26.363551 (+ 1076us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:52:26.365030 (+ 1479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:52:26.365866 (+ 836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:52:26.365981 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:52:26.366132 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0927 14:52:26.366385 (+ 253us) server_negotiation.cc:299] Negotiation successful
0927 14:52:26.366488 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1287,"thread_start_us":154,"threads_started":1}
I20260927 14:52:26.367117 11108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:52:26.360736 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33238 (local address 127.0.98.2:35343)
0927 14:52:26.361373 (+ 637us) server_negotiation.cc:206] Beginning negotiation
0927 14:52:26.361385 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 14:52:26.361690 (+ 305us) server_negotiation.cc:407] Connection header received
0927 14:52:26.361859 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:52:26.361865 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:52:26.362175 (+ 310us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:52:26.362322 (+ 147us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 14:52:26.362758 (+ 436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:52:26.363675 (+ 917us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:52:26.365314 (+ 1639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:52:26.366054 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:52:26.366157 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:52:26.366300 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0927 14:52:26.366893 (+ 593us) server_negotiation.cc:299] Negotiation successful
0927 14:52:26.366984 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":480,"thread_start_us":102,"threads_started":1}
I20260927 14:52:26.367172 11107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:52:26.360693 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50966 (local address 127.0.98.1:43159)
0927 14:52:26.361413 (+ 720us) server_negotiation.cc:206] Beginning negotiation
0927 14:52:26.361428 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 14:52:26.361956 (+ 528us) server_negotiation.cc:407] Connection header received
0927 14:52:26.362165 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:52:26.362172 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:52:26.362337 (+ 165us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:52:26.362444 (+ 107us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 14:52:26.362856 (+ 412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:52:26.364028 (+ 1172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:52:26.365472 (+ 1444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:52:26.366083 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:52:26.366184 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:52:26.366331 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0927 14:52:26.366913 (+ 582us) server_negotiation.cc:299] Negotiation successful
0927 14:52:26.367017 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":623,"thread_start_us":73,"threads_started":1}
I20260927 14:52:26.366822 11105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:52:26.359942 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:43159 (local address 127.0.0.1:50966)
0927 14:52:26.361877 (+ 1935us) negotiation.cc:107] Waiting for socket to connect
0927 14:52:26.361890 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 14:52:26.362020 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:52:26.362621 (+ 601us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:52:26.362623 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:52:26.362636 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 14:52:26.362737 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:52:26.362744 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:52:26.364184 (+ 1440us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:52:26.364187 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:52:26.365361 (+ 1174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:52:26.365367 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:52:26.366391 (+ 1024us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:52:26.366397 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:52:26.366495 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:52:26.366516 (+ 21us) client_negotiation.cc:769] Sending connection context
0927 14:52:26.366586 (+ 70us) client_negotiation.cc:240] Negotiation successful
0927 14:52:26.366671 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":833,"mutex_wait_us":1,"thread_start_us":63,"threads_started":1}
I20260927 14:52:26.367367 11104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:52:26.359973 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:42533 (local address 127.0.0.1:48882)
0927 14:52:26.361082 (+ 1109us) negotiation.cc:107] Waiting for socket to connect
0927 14:52:26.361135 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0927 14:52:26.361286 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:52:26.362193 (+ 907us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:52:26.362199 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:52:26.362231 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 14:52:26.362356 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:52:26.362362 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:52:26.363759 (+ 1397us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:52:26.363762 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:52:26.364912 (+ 1150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:52:26.364921 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:52:26.365996 (+ 1075us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:52:26.366000 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:52:26.366098 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:52:26.366165 (+ 67us) client_negotiation.cc:769] Sending connection context
0927 14:52:26.366270 (+ 105us) client_negotiation.cc:240] Negotiation successful
0927 14:52:26.367241 (+ 971us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":999,"mutex_wait_us":1,"thread_start_us":86,"threads_started":1}
I20260927 14:52:26.370070 10758 tablet_service.cc:1467] Processing CreateTablet for tablet 928e3594947f4fa08d64c5736bbd4048 (DEFAULT_TABLE table=test-table [id=64660fe6efd2498983a5d2fec4a5ff42]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 14:52:26.370168 11026 tablet_service.cc:1467] Processing CreateTablet for tablet 928e3594947f4fa08d64c5736bbd4048 (DEFAULT_TABLE table=test-table [id=64660fe6efd2498983a5d2fec4a5ff42]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 14:52:26.370158 10892 tablet_service.cc:1467] Processing CreateTablet for tablet 928e3594947f4fa08d64c5736bbd4048 (DEFAULT_TABLE table=test-table [id=64660fe6efd2498983a5d2fec4a5ff42]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 14:52:26.371143 10758 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 928e3594947f4fa08d64c5736bbd4048. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 14:52:26.371142 10892 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 928e3594947f4fa08d64c5736bbd4048. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 14:52:26.371147 11026 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 928e3594947f4fa08d64c5736bbd4048. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 14:52:26.377192 11109 tablet_bootstrap.cc:492] T 928e3594947f4fa08d64c5736bbd4048 P ce29d918e07c44a9befb799462f534d4: Bootstrap starting.
I20260927 14:52:26.378638 11110 tablet_bootstrap.cc:492] T 928e3594947f4fa08d64c5736bbd4048 P ba41bd7f6425453da6cf1e9a87425c7b: Bootstrap starting.
I20260927 14:52:26.379410 11109 tablet_bootstrap.cc:654] T 928e3594947f4fa08d64c5736bbd4048 P ce29d918e07c44a9befb799462f534d4: Neither blocks nor log segments found. Creating new log.
I20260927 14:52:26.380535 11109 log.cc:824] T 928e3594947f4fa08d64c5736bbd4048 P ce29d918e07c44a9befb799462f534d4: Log is configured to *not* fsync() on all Append() calls
I20260927 14:52:26.380712 11110 tablet_bootstrap.cc:654] T 928e3594947f4fa08d64c5736bbd4048 P ba41bd7f6425453da6cf1e9a87425c7b: Neither blocks nor log segments found. Creating new log.
I20260927 14:52:26.381129 11111 tablet_bootstrap.cc:492] T 928e3594947f4fa08d64c5736bbd4048 P 5aa838a7144d4b1caa83eab5eab0cdb9: Bootstrap starting.
I20260927 14:52:26.381647 11110 log.cc:824] T 928e3594947f4fa08d64c5736bbd4048 P ba41bd7f6425453da6cf1e9a87425c7b: Log is configured to *not* fsync() on all Append() calls
I20260927 14:52:26.382555 11109 tablet_bootstrap.cc:492] T 928e3594947f4fa08d64c5736bbd4048 P ce29d918e07c44a9befb799462f534d4: No bootstrap required, opened a new log
I20260927 14:52:26.382759 11109 ts_tablet_manager.cc:1397] T 928e3594947f4fa08d64c5736bbd4048 P ce29d918e07c44a9befb799462f534d4: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260927 14:52:26.383257 11111 tablet_bootstrap.cc:654] T 928e3594947f4fa08d64c5736bbd4048 P 5aa838a7144d4b1caa83eab5eab0cdb9: Neither blocks nor log segments found. Creating new log.
I20260927 14:52:26.383750 11110 tablet_bootstrap.cc:492] T 928e3594947f4fa08d64c5736bbd4048 P ba41bd7f6425453da6cf1e9a87425c7b: No bootstrap required, opened a new log
I20260927 14:52:26.383944 11110 ts_tablet_manager.cc:1397] T 928e3594947f4fa08d64c5736bbd4048 P ba41bd7f6425453da6cf1e9a87425c7b: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260927 14:52:26.384506 11111 log.cc:824] T 928e3594947f4fa08d64c5736bbd4048 P 5aa838a7144d4b1caa83eab5eab0cdb9: Log is configured to *not* fsync() on all Append() calls
I20260927 14:52:26.386408 11111 tablet_bootstrap.cc:492] T 928e3594947f4fa08d64c5736bbd4048 P 5aa838a7144d4b1caa83eab5eab0cdb9: No bootstrap required, opened a new log
I20260927 14:52:26.386565 11111 ts_tablet_manager.cc:1397] T 928e3594947f4fa08d64c5736bbd4048 P 5aa838a7144d4b1caa83eab5eab0cdb9: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260927 14:52:26.386372 11109 raft_consensus.cc:348] T 928e3594947f4fa08d64c5736bbd4048 P ce29d918e07c44a9befb799462f534d4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce29d918e07c44a9befb799462f534d4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43159 } } peers { permanent_uuid: "5aa838a7144d4b1caa83eab5eab0cdb9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42533 } } peers { permanent_uuid: "ba41bd7f6425453da6cf1e9a87425c7b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35343 } }
I20260927 14:52:26.386629 11109 raft_consensus.cc:374] T 928e3594947f4fa08d64c5736bbd4048 P ce29d918e07c44a9befb799462f534d4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 14:52:26.386682 11109 raft_consensus.cc:729] T 928e3594947f4fa08d64c5736bbd4048 P ce29d918e07c44a9befb799462f534d4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ce29d918e07c44a9befb799462f534d4, State: Initialized, Role: FOLLOWER
I20260927 14:52:26.387032 11110 raft_consensus.cc:348] T 928e3594947f4fa08d64c5736bbd4048 P ba41bd7f6425453da6cf1e9a87425c7b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce29d918e07c44a9befb799462f534d4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43159 } } peers { permanent_uuid: "5aa838a7144d4b1caa83eab5eab0cdb9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42533 } } peers { permanent_uuid: "ba41bd7f6425453da6cf1e9a87425c7b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35343 } }
I20260927 14:52:26.387130 11109 consensus_queue.cc:260] T 928e3594947f4fa08d64c5736bbd4048 P ce29d918e07c44a9befb799462f534d4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce29d918e07c44a9befb799462f534d4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43159 } } peers { permanent_uuid: "5aa838a7144d4b1caa83eab5eab0cdb9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42533 } } peers { permanent_uuid: "ba41bd7f6425453da6cf1e9a87425c7b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35343 } }
I20260927 14:52:26.387295 11110 raft_consensus.cc:374] T 928e3594947f4fa08d64c5736bbd4048 P ba41bd7f6425453da6cf1e9a87425c7b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 14:52:26.387349 11110 raft_consensus.cc:729] T 928e3594947f4fa08d64c5736bbd4048 P ba41bd7f6425453da6cf1e9a87425c7b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ba41bd7f6425453da6cf1e9a87425c7b, State: Initialized, Role: FOLLOWER
I20260927 14:52:26.387811 11110 consensus_queue.cc:260] T 928e3594947f4fa08d64c5736bbd4048 P ba41bd7f6425453da6cf1e9a87425c7b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce29d918e07c44a9befb799462f534d4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43159 } } peers { permanent_uuid: "5aa838a7144d4b1caa83eab5eab0cdb9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42533 } } peers { permanent_uuid: "ba41bd7f6425453da6cf1e9a87425c7b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35343 } }
I20260927 14:52:26.388088 11109 ts_tablet_manager.cc:1428] T 928e3594947f4fa08d64c5736bbd4048 P ce29d918e07c44a9befb799462f534d4: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260927 14:52:26.388466 10824 heartbeater.cc:503] Master 127.0.98.62:39611 was elected leader, sending a full tablet report...
I20260927 14:52:26.388782 10958 heartbeater.cc:503] Master 127.0.98.62:39611 was elected leader, sending a full tablet report...
I20260927 14:52:26.389384 11110 ts_tablet_manager.cc:1428] T 928e3594947f4fa08d64c5736bbd4048 P ba41bd7f6425453da6cf1e9a87425c7b: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260927 14:52:26.390409 11111 raft_consensus.cc:348] T 928e3594947f4fa08d64c5736bbd4048 P 5aa838a7144d4b1caa83eab5eab0cdb9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce29d918e07c44a9befb799462f534d4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43159 } } peers { permanent_uuid: "5aa838a7144d4b1caa83eab5eab0cdb9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42533 } } peers { permanent_uuid: "ba41bd7f6425453da6cf1e9a87425c7b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35343 } }
I20260927 14:52:26.390707 11111 raft_consensus.cc:374] T 928e3594947f4fa08d64c5736bbd4048 P 5aa838a7144d4b1caa83eab5eab0cdb9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 14:52:26.390781 11111 raft_consensus.cc:729] T 928e3594947f4fa08d64c5736bbd4048 P 5aa838a7144d4b1caa83eab5eab0cdb9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5aa838a7144d4b1caa83eab5eab0cdb9, State: Initialized, Role: FOLLOWER
I20260927 14:52:26.391199 11111 consensus_queue.cc:260] T 928e3594947f4fa08d64c5736bbd4048 P 5aa838a7144d4b1caa83eab5eab0cdb9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce29d918e07c44a9befb799462f534d4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43159 } } peers { permanent_uuid: "5aa838a7144d4b1caa83eab5eab0cdb9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42533 } } peers { permanent_uuid: "ba41bd7f6425453da6cf1e9a87425c7b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35343 } }
I20260927 14:52:26.391883 11092 heartbeater.cc:503] Master 127.0.98.62:39611 was elected leader, sending a full tablet report...
I20260927 14:52:26.392119 11111 ts_tablet_manager.cc:1428] T 928e3594947f4fa08d64c5736bbd4048 P 5aa838a7144d4b1caa83eab5eab0cdb9: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260927 14:52:26.392513 11117 raft_consensus.cc:482] T 928e3594947f4fa08d64c5736bbd4048 P 5aa838a7144d4b1caa83eab5eab0cdb9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 14:52:26.392647 11117 raft_consensus.cc:504] T 928e3594947f4fa08d64c5736bbd4048 P 5aa838a7144d4b1caa83eab5eab0cdb9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce29d918e07c44a9befb799462f534d4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43159 } } peers { permanent_uuid: "5aa838a7144d4b1caa83eab5eab0cdb9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42533 } } peers { permanent_uuid: "ba41bd7f6425453da6cf1e9a87425c7b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35343 } }
I20260927 14:52:26.393833 11117 leader_election.cc:288] T 928e3594947f4fa08d64c5736bbd4048 P 5aa838a7144d4b1caa83eab5eab0cdb9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ce29d918e07c44a9befb799462f534d4 (127.0.98.1:43159), ba41bd7f6425453da6cf1e9a87425c7b (127.0.98.2:35343)
I20260927 14:52:26.397689 11107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:52:26.394117 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:54975 (local address 127.0.98.1:43159)
0927 14:52:26.394257 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0927 14:52:26.394263 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 14:52:26.394279 (+ 16us) server_negotiation.cc:407] Connection header received
0927 14:52:26.394360 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:52:26.394362 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:52:26.394395 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:52:26.394458 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 14:52:26.394730 (+ 272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:52:26.395291 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:52:26.396771 (+ 1480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:52:26.397229 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:52:26.397311 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:52:26.397358 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0927 14:52:26.397553 (+ 195us) server_negotiation.cc:299] Negotiation successful
0927 14:52:26.397599 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260927 14:52:26.397730 11094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:52:26.394036 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:43159 (local address 127.0.98.3:54975)
0927 14:52:26.394175 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0927 14:52:26.394191 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 14:52:26.394268 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:52:26.394489 (+ 221us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:52:26.394492 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:52:26.394512 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 14:52:26.394636 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:52:26.394642 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:52:26.395402 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:52:26.395408 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:52:26.396664 (+ 1256us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:52:26.396673 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:52:26.397334 (+ 661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:52:26.397338 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:52:26.397440 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:52:26.397464 (+ 24us) client_negotiation.cc:769] Sending connection context
0927 14:52:26.397545 (+ 81us) client_negotiation.cc:240] Negotiation successful
0927 14:52:26.397612 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":44}
I20260927 14:52:26.398360 10778 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "928e3594947f4fa08d64c5736bbd4048" candidate_uuid: "5aa838a7144d4b1caa83eab5eab0cdb9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ce29d918e07c44a9befb799462f534d4" is_pre_election: true
I20260927 14:52:26.398612 10778 raft_consensus.cc:2454] T 928e3594947f4fa08d64c5736bbd4048 P ce29d918e07c44a9befb799462f534d4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5aa838a7144d4b1caa83eab5eab0cdb9 in term 0.
I20260927 14:52:26.399066 11108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:52:26.394823 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:43737 (local address 127.0.98.2:35343)
0927 14:52:26.394949 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0927 14:52:26.394954 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 14:52:26.395427 (+ 473us) server_negotiation.cc:407] Connection header received
0927 14:52:26.395591 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:52:26.395595 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:52:26.395644 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:52:26.395750 (+ 106us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 14:52:26.396018 (+ 268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:52:26.396727 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:52:26.398049 (+ 1322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:52:26.398563 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:52:26.398646 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:52:26.398687 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0927 14:52:26.398895 (+ 208us) server_negotiation.cc:299] Negotiation successful
0927 14:52:26.398980 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260927 14:52:26.399088 11118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:52:26.394463 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:35343 (local address 127.0.98.3:43737)
0927 14:52:26.395351 (+ 888us) negotiation.cc:107] Waiting for socket to connect
0927 14:52:26.395365 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 14:52:26.395482 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:52:26.395761 (+ 279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:52:26.395764 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:52:26.395782 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 14:52:26.395890 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:52:26.395895 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:52:26.396854 (+ 959us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:52:26.396857 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:52:26.397912 (+ 1055us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:52:26.397922 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:52:26.398679 (+ 757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:52:26.398685 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:52:26.398775 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:52:26.398800 (+ 25us) client_negotiation.cc:769] Sending connection context
0927 14:52:26.398877 (+ 77us) client_negotiation.cc:240] Negotiation successful
0927 14:52:26.398956 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":823,"thread_start_us":112,"threads_started":1}
I20260927 14:52:26.399046 10980 leader_election.cc:302] T 928e3594947f4fa08d64c5736bbd4048 P 5aa838a7144d4b1caa83eab5eab0cdb9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5aa838a7144d4b1caa83eab5eab0cdb9, ce29d918e07c44a9befb799462f534d4; no voters:
I20260927 14:52:26.399641 10912 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "928e3594947f4fa08d64c5736bbd4048" candidate_uuid: "5aa838a7144d4b1caa83eab5eab0cdb9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ba41bd7f6425453da6cf1e9a87425c7b" is_pre_election: true
I20260927 14:52:26.399884 10912 raft_consensus.cc:2454] T 928e3594947f4fa08d64c5736bbd4048 P ba41bd7f6425453da6cf1e9a87425c7b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5aa838a7144d4b1caa83eab5eab0cdb9 in term 0.
I20260927 14:52:26.400156 11117 raft_consensus.cc:2781] T 928e3594947f4fa08d64c5736bbd4048 P 5aa838a7144d4b1caa83eab5eab0cdb9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 14:52:26.400244 11117 raft_consensus.cc:482] T 928e3594947f4fa08d64c5736bbd4048 P 5aa838a7144d4b1caa83eab5eab0cdb9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 14:52:26.400290 11117 raft_consensus.cc:3037] T 928e3594947f4fa08d64c5736bbd4048 P 5aa838a7144d4b1caa83eab5eab0cdb9 [term 0 FOLLOWER]: Advancing to term 1
I20260927 14:52:26.401374 11117 raft_consensus.cc:504] T 928e3594947f4fa08d64c5736bbd4048 P 5aa838a7144d4b1caa83eab5eab0cdb9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce29d918e07c44a9befb799462f534d4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43159 } } peers { permanent_uuid: "5aa838a7144d4b1caa83eab5eab0cdb9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42533 } } peers { permanent_uuid: "ba41bd7f6425453da6cf1e9a87425c7b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35343 } }
I20260927 14:52:26.401674 11117 leader_election.cc:288] T 928e3594947f4fa08d64c5736bbd4048 P 5aa838a7144d4b1caa83eab5eab0cdb9 [CANDIDATE]: Term 1 election: Requested vote from peers ce29d918e07c44a9befb799462f534d4 (127.0.98.1:43159), ba41bd7f6425453da6cf1e9a87425c7b (127.0.98.2:35343)
I20260927 14:52:26.401979 10778 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "928e3594947f4fa08d64c5736bbd4048" candidate_uuid: "5aa838a7144d4b1caa83eab5eab0cdb9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ce29d918e07c44a9befb799462f534d4"
I20260927 14:52:26.401979 10912 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "928e3594947f4fa08d64c5736bbd4048" candidate_uuid: "5aa838a7144d4b1caa83eab5eab0cdb9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ba41bd7f6425453da6cf1e9a87425c7b"
I20260927 14:52:26.402089 10912 raft_consensus.cc:3037] T 928e3594947f4fa08d64c5736bbd4048 P ba41bd7f6425453da6cf1e9a87425c7b [term 0 FOLLOWER]: Advancing to term 1
I20260927 14:52:26.402114 10778 raft_consensus.cc:3037] T 928e3594947f4fa08d64c5736bbd4048 P ce29d918e07c44a9befb799462f534d4 [term 0 FOLLOWER]: Advancing to term 1
I20260927 14:52:26.403308 10912 raft_consensus.cc:2454] T 928e3594947f4fa08d64c5736bbd4048 P ba41bd7f6425453da6cf1e9a87425c7b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5aa838a7144d4b1caa83eab5eab0cdb9 in term 1.
I20260927 14:52:26.403308 10778 raft_consensus.cc:2454] T 928e3594947f4fa08d64c5736bbd4048 P ce29d918e07c44a9befb799462f534d4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5aa838a7144d4b1caa83eab5eab0cdb9 in term 1.
I20260927 14:52:26.403613 10982 leader_election.cc:302] T 928e3594947f4fa08d64c5736bbd4048 P 5aa838a7144d4b1caa83eab5eab0cdb9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5aa838a7144d4b1caa83eab5eab0cdb9, ba41bd7f6425453da6cf1e9a87425c7b; no voters:
I20260927 14:52:26.403777 11117 raft_consensus.cc:2781] T 928e3594947f4fa08d64c5736bbd4048 P 5aa838a7144d4b1caa83eab5eab0cdb9 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 14:52:26.403959 11117 raft_consensus.cc:686] T 928e3594947f4fa08d64c5736bbd4048 P 5aa838a7144d4b1caa83eab5eab0cdb9 [term 1 LEADER]: Becoming Leader. State: Replica: 5aa838a7144d4b1caa83eab5eab0cdb9, State: Running, Role: LEADER
I20260927 14:52:26.404273 11117 consensus_queue.cc:237] T 928e3594947f4fa08d64c5736bbd4048 P 5aa838a7144d4b1caa83eab5eab0cdb9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce29d918e07c44a9befb799462f534d4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43159 } } peers { permanent_uuid: "5aa838a7144d4b1caa83eab5eab0cdb9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42533 } } peers { permanent_uuid: "ba41bd7f6425453da6cf1e9a87425c7b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35343 } }
I20260927 14:52:26.407508 10631 catalog_manager.cc:5495] T 928e3594947f4fa08d64c5736bbd4048 P 5aa838a7144d4b1caa83eab5eab0cdb9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5aa838a7144d4b1caa83eab5eab0cdb9 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "5aa838a7144d4b1caa83eab5eab0cdb9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce29d918e07c44a9befb799462f534d4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43159 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5aa838a7144d4b1caa83eab5eab0cdb9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42533 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ba41bd7f6425453da6cf1e9a87425c7b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35343 } health_report { overall_health: UNKNOWN } } }
I20260927 14:52:26.427541 11106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:52:26.424164 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48892 (local address 127.0.98.3:42533)
0927 14:52:26.424281 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0927 14:52:26.424285 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 14:52:26.424297 (+ 12us) server_negotiation.cc:407] Connection header received
0927 14:52:26.424438 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:52:26.424440 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:52:26.424495 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:52:26.424582 (+ 87us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 14:52:26.424820 (+ 238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:52:26.425383 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:52:26.426214 (+ 831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:52:26.426415 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:52:26.426489 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:52:26.426778 (+ 289us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 14:52:26.427180 (+ 402us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 14:52:26.427269 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0927 14:52:26.427391 (+ 122us) server_negotiation.cc:299] Negotiation successful
0927 14:52:26.427435 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260927 14:52:26.433897 10778 raft_consensus.cc:1261] T 928e3594947f4fa08d64c5736bbd4048 P ce29d918e07c44a9befb799462f534d4 [term 1 FOLLOWER]: Refusing update from remote peer 5aa838a7144d4b1caa83eab5eab0cdb9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 14:52:26.433897 10912 raft_consensus.cc:1261] T 928e3594947f4fa08d64c5736bbd4048 P ba41bd7f6425453da6cf1e9a87425c7b [term 1 FOLLOWER]: Refusing update from remote peer 5aa838a7144d4b1caa83eab5eab0cdb9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 14:52:26.434689 11119 consensus_queue.cc:1035] T 928e3594947f4fa08d64c5736bbd4048 P 5aa838a7144d4b1caa83eab5eab0cdb9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce29d918e07c44a9befb799462f534d4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43159 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 14:52:26.434921 11117 consensus_queue.cc:1035] T 928e3594947f4fa08d64c5736bbd4048 P 5aa838a7144d4b1caa83eab5eab0cdb9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ba41bd7f6425453da6cf1e9a87425c7b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35343 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 14:52:26.452243 10631 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55244:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 14:52:26.452435 10631 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55244:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 14:52:26.454861 10631 catalog_manager.cc:5782] T 00000000000000000000000000000000 P cd0e61e12fc94075901d6599bb52df98: Sending DeleteTablet for 3 replicas of tablet 928e3594947f4fa08d64c5736bbd4048
I20260927 14:52:26.455613 11026 tablet_service.cc:1514] Processing DeleteTablet for tablet 928e3594947f4fa08d64c5736bbd4048 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 14:52:26 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:48882
I20260927 14:52:26.456116 11132 tablet_replica.cc:330] T 928e3594947f4fa08d64c5736bbd4048 P 5aa838a7144d4b1caa83eab5eab0cdb9: stopping tablet replica
I20260927 14:52:26.456156 10758 tablet_service.cc:1514] Processing DeleteTablet for tablet 928e3594947f4fa08d64c5736bbd4048 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 14:52:26 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:50966
I20260927 14:52:26.456259 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6936m5/build/debug/bin/kudu with pid 10695
I20260927 14:52:26.456599 11132 raft_consensus.cc:2229] T 928e3594947f4fa08d64c5736bbd4048 P 5aa838a7144d4b1caa83eab5eab0cdb9 [term 1 LEADER]: Raft consensus shutting down.
I20260927 14:52:26.457401 11132 raft_consensus.cc:2258] T 928e3594947f4fa08d64c5736bbd4048 P 5aa838a7144d4b1caa83eab5eab0cdb9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 14:52:26.457883 10892 tablet_service.cc:1514] Processing DeleteTablet for tablet 928e3594947f4fa08d64c5736bbd4048 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 14:52:26 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:33238
I20260927 14:52:26.458716 11133 tablet_replica.cc:330] T 928e3594947f4fa08d64c5736bbd4048 P ba41bd7f6425453da6cf1e9a87425c7b: stopping tablet replica
I20260927 14:52:26.459009 11133 raft_consensus.cc:2229] T 928e3594947f4fa08d64c5736bbd4048 P ba41bd7f6425453da6cf1e9a87425c7b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 14:52:26.459281 11132 ts_tablet_manager.cc:1905] T 928e3594947f4fa08d64c5736bbd4048 P 5aa838a7144d4b1caa83eab5eab0cdb9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 14:52:26.459805 11133 raft_consensus.cc:2258] T 928e3594947f4fa08d64c5736bbd4048 P ba41bd7f6425453da6cf1e9a87425c7b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 14:52:26.461272 11133 ts_tablet_manager.cc:1905] T 928e3594947f4fa08d64c5736bbd4048 P ba41bd7f6425453da6cf1e9a87425c7b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 14:52:26.462366 11132 ts_tablet_manager.cc:1918] T 928e3594947f4fa08d64c5736bbd4048 P 5aa838a7144d4b1caa83eab5eab0cdb9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 14:52:26.462531 11132 log.cc:1196] T 928e3594947f4fa08d64c5736bbd4048 P 5aa838a7144d4b1caa83eab5eab0cdb9: Deleting WAL directory at /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790520684602251-392-0/minicluster-data/ts-2/wal/wals/928e3594947f4fa08d64c5736bbd4048
I20260927 14:52:26.462955 11132 ts_tablet_manager.cc:1939] T 928e3594947f4fa08d64c5736bbd4048 P 5aa838a7144d4b1caa83eab5eab0cdb9: Deleting consensus metadata
I20260927 14:52:26.464046 11133 ts_tablet_manager.cc:1918] T 928e3594947f4fa08d64c5736bbd4048 P ba41bd7f6425453da6cf1e9a87425c7b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 14:52:26.464149 11133 log.cc:1196] T 928e3594947f4fa08d64c5736bbd4048 P ba41bd7f6425453da6cf1e9a87425c7b: Deleting WAL directory at /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790520684602251-392-0/minicluster-data/ts-1/wal/wals/928e3594947f4fa08d64c5736bbd4048
I20260927 14:52:26.464497 11133 ts_tablet_manager.cc:1939] T 928e3594947f4fa08d64c5736bbd4048 P ba41bd7f6425453da6cf1e9a87425c7b: Deleting consensus metadata
I20260927 14:52:26.464509 10619 catalog_manager.cc:4841] TS 5aa838a7144d4b1caa83eab5eab0cdb9 (127.0.98.3:42533): tablet 928e3594947f4fa08d64c5736bbd4048 (table test-table [id=64660fe6efd2498983a5d2fec4a5ff42]) successfully deleted
W20260927 14:52:26.465544 10618 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:43159 (error 108)
I20260927 14:52:26.465909 10620 catalog_manager.cc:4841] TS ba41bd7f6425453da6cf1e9a87425c7b (127.0.98.2:35343): tablet 928e3594947f4fa08d64c5736bbd4048 (table test-table [id=64660fe6efd2498983a5d2fec4a5ff42]) successfully deleted
I20260927 14:52:26.466611 11104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:52:26.466303 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:43159 (local address 127.0.0.1:50970)
0927 14:52:26.466449 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0927 14:52:26.466518 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:43159: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260927 14:52:26.466796 10618 catalog_manager.cc:4584] TS ce29d918e07c44a9befb799462f534d4 (127.0.98.1:43159): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 928e3594947f4fa08d64c5736bbd4048: Network error: Client connection negotiation failed: client connection to 127.0.98.1:43159: connect: Connection refused (error 111)
I20260927 14:52:26.467464 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6936m5/build/debug/bin/kudu with pid 10829
I20260927 14:52:26.474226 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6936m5/build/debug/bin/kudu with pid 10963
I20260927 14:52:26.478808 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6936m5/build/debug/bin/kudu with pid 10601
2026-09-27T14:52:26Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (2193 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790520684602251-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 14:52:26 dist-test-slave-xk1g krb5kdc[11137](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 14:52:26 dist-test-slave-xk1g krb5kdc[11137](info): set up 2 sockets
Sep 27 14:52:26 dist-test-slave-xk1g krb5kdc[11137](info): commencing operation
krb5kdc: starting...
W20260927 14:52:27.522197 392 mini_kdc.cc:121] Time spent starting KDC: real 1.022s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790520684602251-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790520684602251-392-0/krb5kdc/test-user.keytab.
Sep 27 14:52:27 dist-test-slave-xk1g krb5kdc[11137](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790520747, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T14:52:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T14:52:27Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790520684602251-392-0/krb5kdc/rangeradmin_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790520684602251-392-0/krb5kdc/rangeradmin_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790520684602251-392-0/krb5kdc/rangerlookup_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790520684602251-392-0/krb5kdc/rangerlookup_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790520684602251-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790520684602251-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
I20260927 14:52:27.691779 392 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790520684602251-392-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-task6936m5/build/debug/bin/postgres/pg_ctl -D /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790520684602251-392-0/minicluster-data/postgres -l logfile start
2026-09-27 14:52:30.212 UTC [11167] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-27 14:52:30.212 UTC [11167] LOG: listening on IPv4 address "127.0.98.20", port 42649
2026-09-27 14:52:30.216 UTC [11167] LOG: listening on Unix socket "/tmp/.s.PGSQL.42649"
2026-09-27 14:52:30.250 UTC [11170] LOG: database system was shut down at 2026-09-27 14:52:28 UTC
2026-09-27 14:52:30.254 UTC [11167] LOG: database system is ready to accept connections
I20260927 14:52:31.241806 392 mini_postgres.cc:96] Postgres bound to 42649
2026-09-27 14:52:31.248 UTC [11178] FATAL: database "slave" does not exist
127.0.98.20:42649 - accepting connections
I20260927 14:52:31.248731 392 mini_ranger.cc:161] Starting Ranger...
I20260927 14:52:31.263116 392 mini_ranger.cc:84] Created miniranger Postgres user
I20260927 14:52:31.752877 392 mini_ranger.cc:87] Created ranger Postgres database
I20260927 14:52:31.752955 392 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790520684602251-392-0/minicluster-data/ranger-admin
2026-09-27 14:52:32,024 [I] DB FLAVOR :POSTGRES
2026-09-27 14:52:32,024 [I] --------- Verifying Ranger DB connection ---------
2026-09-27 14:52:32,025 [I] Checking connection..
2026-09-27 14:52:32,025 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6936m5/build/debug/bin/postgresql.jar:/tmp/dist-test-task6936m5/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:42649/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-27 14:52:32,264 [I] Checking connection passed.
2026-09-27 14:52:32,265 [I] --------- Verifying version history table ---------
2026-09-27 14:52:32,265 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6936m5/build/debug/bin/postgresql.jar:/tmp/dist-test-task6936m5/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:42649/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-27 14:52:32,493 [I] Table x_db_version_h does not exist in database ranger
2026-09-27 14:52:32,493 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6936m5/build/debug/bin/postgresql.jar:/tmp/dist-test-task6936m5/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:42649/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-27 14:52:32,744 [I] Table x_db_version_h does not exist in database ranger
2026-09-27 14:52:32,744 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-27 14:52:32,744 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6936m5/build/debug/bin/postgresql.jar:/tmp/dist-test-task6936m5/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:42649/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task6936m5/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-27 14:52:32.982 UTC [11242] WARNING: there is no transaction in progress
2026-09-27 14:52:32,988 [I] create_dbversion_catalog.sql file imported successfully
2026-09-27 14:52:32,989 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6936m5/build/debug/bin/postgresql.jar:/tmp/dist-test-task6936m5/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:42649/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-27 14:52:33,222 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-27 14:52:33,223 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-27 14:52:33,223 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6936m5/build/debug/bin/postgresql.jar:/tmp/dist-test-task6936m5/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:42649/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-27 14:52:33,449 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6936m5/build/debug/bin/postgresql.jar:/tmp/dist-test-task6936m5/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:42649/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-27 14:52:33,673 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6936m5/build/debug/bin/postgresql.jar:/tmp/dist-test-task6936m5/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:42649/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-xk1g.c.gcp-upstream.internal','N') ;"
2026-09-27 14:52:33,906 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6936m5/build/debug/bin/postgresql.jar:/tmp/dist-test-task6936m5/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:42649/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-27 14:52:34,131 [I] Table x_portal_user does not exist in database ranger
2026-09-27 14:52:34,131 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6936m5/build/debug/bin/postgresql.jar:/tmp/dist-test-task6936m5/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:42649/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-27 14:52:34,367 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-27 14:52:34,367 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6936m5/build/debug/bin/postgresql.jar:/tmp/dist-test-task6936m5/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:42649/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-27 14:52:34,593 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6936m5/build/debug/bin/postgresql.jar:/tmp/dist-test-task6936m5/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:42649/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-27 14:52:34,850 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-27 14:52:34,850 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6936m5/build/debug/bin/postgresql.jar:/tmp/dist-test-task6936m5/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:42649/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task6936m5/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-27 14:52:35.806 UTC [11350] WARNING: there is no transaction in progress
2026-09-27 14:52:35.822 UTC [11350] WARNING: there is no transaction in progress
2026-09-27 14:52:35.838 UTC [11350] WARNING: there is no transaction in progress
2026-09-27 14:52:35.852 UTC [11350] WARNING: there is no transaction in progress
2026-09-27 14:52:35.869 UTC [11350] WARNING: there is no transaction in progress
2026-09-27 14:52:35.888 UTC [11350] WARNING: there is no transaction in progress
2026-09-27 14:52:35.944 UTC [11350] WARNING: there is no transaction in progress
2026-09-27 14:52:35.951 UTC [11350] WARNING: there is no transaction in progress
2026-09-27 14:52:35.965 UTC [11350] WARNING: there is no transaction in progress
2026-09-27 14:52:35.975 UTC [11350] WARNING: there is no transaction in progress
2026-09-27 14:52:35.985 UTC [11350] WARNING: there is no transaction in progress
2026-09-27 14:52:35.999 UTC [11350] WARNING: there is no transaction in progress
2026-09-27 14:52:36.009 UTC [11350] WARNING: there is no transaction in progress
2026-09-27 14:52:36.016 UTC [11350] WARNING: there is no transaction in progress
2026-09-27 14:52:36.023 UTC [11350] WARNING: there is no transaction in progress
2026-09-27 14:52:36.030 UTC [11350] WARNING: there is no transaction in progress
2026-09-27 14:52:36,664 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-27 14:52:36,665 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6936m5/build/debug/bin/postgresql.jar:/tmp/dist-test-task6936m5/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:42649/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-27 14:52:36,912 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-27 14:52:36,913 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6936m5/build/debug/bin/postgresql.jar:/tmp/dist-test-task6936m5/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:42649/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-27 14:52:37,142 [I] Table x_portal_user already exists in database 'ranger'
2026-09-27 14:52:37,142 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6936m5/build/debug/bin/postgresql.jar:/tmp/dist-test-task6936m5/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:42649/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-27 14:52:37,392 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-27 14:52:37,392 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6936m5/build/debug/bin/postgresql.jar:/tmp/dist-test-task6936m5/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:42649/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-27 14:52:37,640 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6936m5/build/debug/bin/postgresql.jar:/tmp/dist-test-task6936m5/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:42649/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-27 14:52:37,871 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6936m5/build/debug/bin/postgresql.jar:/tmp/dist-test-task6936m5/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:42649/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-xk1g.c.gcp-upstream.internal';"
2026-09-27 14:52:38,119 [I] CORE_DB_SCHEMA import status has been updated
2026-09-27 14:52:38,120 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6936m5/build/debug/bin/postgresql.jar:/tmp/dist-test-task6936m5/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:42649/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-27 14:52:38,363 [I] DB_PATCHES have already been applied
I20260927 14:52:38.369294 392 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790520684602251-392-0/minicluster-data/ranger-admin:/tmp/dist-test-task6936m5/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-task6936m5/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-task6936m5/build/debug/bin/postgresql.jar:/tmp/dist-test-task6936m5/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260927 14:52:38.369360 392 mini_ranger.cc:193] Using host: 127.0.98.20
I20260927 14:52:38.373042 392 mini_ranger.cc:227] Ranger admin URL: http://127.0.98.20:52621
Sep 27, 2026 2:52:38 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 27, 2026 2:52:38 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 27, 2026 2:52:38 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790520684602251-392-0/minicluster-data/ranger-admin/ews/webapp
Sep 27, 2026 2:52:38 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790520684602251-392-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 27, 2026 2:52:38 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790520684602251-392-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 27, 2026 2:52:39 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790520684602251-392-0/minicluster-data/ranger-admin/ews/webapp].
Sep 27, 2026 2:52:39 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260927 14:52:39.471454 392 mini_ranger.cc:160] Time spent starting Ranger: real 8.223s user 0.005s sys 0.005s
Sep 27, 2026 2:52:59 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 27, 2026 2:52:59 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Sep 27, 2026 2:52:59 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 27, 2026 2:52:59 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 27, 2026 2:52:59 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 27, 2026 2:52:59 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 27, 2026 2:52:59 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 27, 2026 2:52:59 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 27, 2026 2:52:59 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 27, 2026 2:52:59 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 27, 2026 2:52:59 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 27, 2026 2:52:59 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 27, 2026 2:52:59 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 27, 2026 2:52:59 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 27, 2026 2:52:59 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 27, 2026 2:52:59 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 27, 2026 2:52:59 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 27, 2026 2:53:00 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260927 14:53:01.386718 392 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790520684602251-392-0/krb5kdc/rangerkms_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790520684602251-392-0/krb5kdc/rangerkms_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.98.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.0.98.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790520684602251-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790520684602251-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 27 14:53:01 dist-test-slave-xk1g krb5kdc[11137](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790520781, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260927 14:53:01.476143 392 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260927 14:53:01.485162 392 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260927 14:53:02.355284 392 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260927 14:53:02.355389 392 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790520684602251-392-0/minicluster-data/ranger-kms
I20260927 14:53:02.355489 392 mini_ranger_kms.cc:227] Using postgres at 127.0.98.20:42649
2026-09-27 14:53:02,532 [I] DB FLAVOR :POSTGRES
2026-09-27 14:53:02,533 [I] --------- Verifying Ranger DB connection ---------
2026-09-27 14:53:02,533 [I] Checking connection
2026-09-27 14:53:02,533 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6936m5/build/debug/bin/postgresql.jar:/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790520684602251-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:42649/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-27 14:53:02,776 [I] connection success
2026-09-27 14:53:02,777 [I] --------- Verifying Ranger DB tables ---------
2026-09-27 14:53:02,777 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-27 14:53:02,777 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6936m5/build/debug/bin/postgresql.jar:/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790520684602251-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:42649/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-27 14:53:03,038 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-27 14:53:03,039 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-27 14:53:03,039 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-27 14:53:03,039 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6936m5/build/debug/bin/postgresql.jar:/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790520684602251-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:42649/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790520684602251-392-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-27 14:53:03,303 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260927 14:53:03.783207 392 mini_ranger_kms.cc:310] Created kms service
I20260927 14:53:03.983304 392 mini_ranger_kms.cc:326] Created kudu user
I20260927 14:53:04.026291 392 mini_ranger_kms.cc:343] Created rangerkms user
Sep 27 14:53:04 dist-test-slave-xk1g krb5kdc[11137](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790520781, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260927 14:53:04.415122 392 mini_ranger_kms.cc:384] Added ranger policy
I20260927 14:53:04.415253 392 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790520684602251-392-0/minicluster-data/ranger-kms:/tmp/dist-test-task6936m5/build/debug/bin/postgresql.jar:/tmp/dist-test-task6936m5/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-task6936m5/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260927 14:53:04.415275 392 mini_ranger_kms.cc:240] Using host: 127.0.98.20
I20260927 14:53:04.418692 392 mini_ranger_kms.cc:276] Ranger KMS PID: 11530
I20260927 14:53:04.418828 392 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.0.98.20:58651
Sep 27, 2026 2:53:05 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 27, 2026 2:53:05 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 27, 2026 2:53:05 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790520684602251-392-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 27, 2026 2:53:05 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790520684602251-392-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 27, 2026 2:53:06 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790520684602251-392-0/minicluster-data/ranger-kms/ews/webapp].
Sep 27, 2026 2:53:06 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260927 14:53:06.657195 392 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.181s user 0.000s sys 0.162s
I20260927 14:53:06.657337 392 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260927 14:53:06.657444 392 mini_ranger_kms.cc:401] 127.0.98.20:58651/kms/v1/keys
Sep 27 14:53:06 dist-test-slave-xk1g krb5kdc[11137](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790520786, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 27 14:53:07 dist-test-slave-xk1g krb5kdc[11137](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790520786, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
Sep 27 14:53:07 dist-test-slave-xk1g krb5kdc[11137](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790520786, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
Sep 27 14:53:09 dist-test-slave-xk1g krb5kdc[11137](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790520789, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790520684602251-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790520684602251-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 14:53:09.784090 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6936m5/build/debug/bin/kudu
/tmp/dist-test-task6936m5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790520684602251-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790520684602251-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790520684602251-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790520684602251-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41989
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790520684602251-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36119
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:41989
--ranger_config_path=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790520684602251-392-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:58651/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790520684602251-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790520684602251-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790520684602251-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 14:53:09.906927 11581 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 14:53:09.907191 11581 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 14:53:09.907230 11581 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 14:53:09.911259 11581 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 14:53:09.911312 11581 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 14:53:09.911331 11581 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 14:53:09.911347 11581 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 14:53:09.911363 11581 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 14:53:09.916702 11581 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36119
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790520684602251-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790520684602251-392-0/minicluster-data/master-0/wal
--ranger_kms_url=127.0.98.20:58651/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:41989
--ranger_config_path=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790520684602251-392-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790520684602251-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41989
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790520684602251-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11581
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790520684602251-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:15:17 UTC on ccycloud.quasar-ledxfe.root.comops.site
build id 4676134
I20260927 14:53:09.917781 11581 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 14:53:09.918773 11581 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 14:53:09.928037 11589 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 14:53:09.928082 11586 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 14:53:09.928211 11581 server_base.cc:1034] running on GCE node
W20260927 14:53:09.928289 11587 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 14:53:09.929010 11581 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 14:53:09.930154 11581 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 14:53:09.931325 11581 hybrid_clock.cc:648] HybridClock initialized: now 1790520789931293 us; error 47 us; skew 500 ppm
Sep 27 14:53:09 dist-test-slave-xk1g krb5kdc[11137](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790520789, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 14:53:09.934087 11581 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 14:53:09.935045 11581 webserver.cc:460] Webserver started at http://127.0.98.62:36229/ using document root <none> and password file <none>
I20260927 14:53:09.935572 11581 fs_manager.cc:359] Metadata directory not provided
I20260927 14:53:09.935626 11581 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 14:53:09.935832 11581 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 27 14:53:09 dist-test-slave-xk1g krb5kdc[11137](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790520789, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260927 14:53:09.980624 11581 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790520684602251-392-0/minicluster-data/master-0/data/instance:
uuid: "31f00e587ac6439aa5166ae2930cdef0"
format_stamp: "Formatted at 2026-09-27 14:53:09 on dist-test-slave-xk1g"
server_key: "889dcaea6b456966d65d4c689d68dc4f"
server_key_iv: "2c86d24e4ed6255ba1bcc5d57607a15e"
server_key_version: "kuduclusterkey@0"
I20260927 14:53:09.981346 11581 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790520684602251-392-0/minicluster-data/master-0/wal/instance:
uuid: "31f00e587ac6439aa5166ae2930cdef0"
format_stamp: "Formatted at 2026-09-27 14:53:09 on dist-test-slave-xk1g"
server_key: "889dcaea6b456966d65d4c689d68dc4f"
server_key_iv: "2c86d24e4ed6255ba1bcc5d57607a15e"
server_key_version: "kuduclusterkey@0"
I20260927 14:53:09.986323 11581 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.002s
I20260927 14:53:10.001274 11596 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 14:53:10.003054 11581 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260927 14:53:10.003216 11581 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790520684602251-392-0/minicluster-data/master-0/data,/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790520684602251-392-0/minicluster-data/master-0/wal
uuid: "31f00e587ac6439aa5166ae2930cdef0"
format_stamp: "Formatted at 2026-09-27 14:53:09 on dist-test-slave-xk1g"
server_key: "889dcaea6b456966d65d4c689d68dc4f"
server_key_iv: "2c86d24e4ed6255ba1bcc5d57607a15e"
server_key_version: "kuduclusterkey@0"
I20260927 14:53:10.003366 11581 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790520684602251-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790520684602251-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790520684602251-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 14:53:10.028786 11581 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 14:53:10.032130 11581 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 14:53:10.032332 11581 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 14:53:10.040599 11581 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:41989
I20260927 14:53:10.040660 11648 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:41989 every 8 connection(s)
I20260927 14:53:10.041685 11581 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790520684602251-392-0/minicluster-data/master-0/data/info.pb
I20260927 14:53:10.042088 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6936m5/build/debug/bin/kudu as pid 11581
I20260927 14:53:10.042198 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790520684602251-392-0/minicluster-data/master-0/wal/instance
I20260927 14:53:10.042512 392 external_mini_cluster.cc:1428] Setting key a2b7e0c0416f434cfc776642b742f665
I20260927 14:53:10.044893 11649 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 14:53:10.051126 11649 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 31f00e587ac6439aa5166ae2930cdef0: Bootstrap starting.
Sep 27 14:53:10 dist-test-slave-xk1g krb5kdc[11137](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790520789, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 14:53:10.053910 11649 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 31f00e587ac6439aa5166ae2930cdef0: Neither blocks nor log segments found. Creating new log.
I20260927 14:53:10.054785 11649 log.cc:824] T 00000000000000000000000000000000 P 31f00e587ac6439aa5166ae2930cdef0: Log is configured to *not* fsync() on all Append() calls
I20260927 14:53:10.057199 11649 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 31f00e587ac6439aa5166ae2930cdef0: No bootstrap required, opened a new log
I20260927 14:53:10.060051 11652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:53:10.044097 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32816 (local address 127.0.98.62:41989)
0927 14:53:10.044579 (+ 482us) server_negotiation.cc:206] Beginning negotiation
0927 14:53:10.044600 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0927 14:53:10.044916 (+ 316us) server_negotiation.cc:407] Connection header received
0927 14:53:10.045947 (+ 1031us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:53:10.045973 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:53:10.046399 (+ 426us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:53:10.046775 (+ 376us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:53:10.047659 (+ 884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:53:10.049315 (+ 1656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:53:10.050522 (+ 1207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:53:10.051235 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:53:10.051391 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:53:10.054590 (+ 3199us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:53:10.054597 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:53:10.054613 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 14:53:10.054649 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:53:10.057177 (+ 2528us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:53:10.057651 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:53:10.057659 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:53:10.057667 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:53:10.057802 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:53:10.058096 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:53:10.058100 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:53:10.058102 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:53:10.058505 (+ 403us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:53:10.058719 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0927 14:53:10.059028 (+ 309us) server_negotiation.cc:299] Negotiation successful
0927 14:53:10.059333 (+ 305us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":329,"thread_start_us":159,"threads_started":1}
I20260927 14:53:10.061307 11649 raft_consensus.cc:348] T 00000000000000000000000000000000 P 31f00e587ac6439aa5166ae2930cdef0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31f00e587ac6439aa5166ae2930cdef0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41989 } }
I20260927 14:53:10.061589 11649 raft_consensus.cc:374] T 00000000000000000000000000000000 P 31f00e587ac6439aa5166ae2930cdef0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 14:53:10.061671 11649 raft_consensus.cc:729] T 00000000000000000000000000000000 P 31f00e587ac6439aa5166ae2930cdef0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31f00e587ac6439aa5166ae2930cdef0, State: Initialized, Role: FOLLOWER
I20260927 14:53:10.062153 11649 consensus_queue.cc:260] T 00000000000000000000000000000000 P 31f00e587ac6439aa5166ae2930cdef0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31f00e587ac6439aa5166ae2930cdef0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41989 } }
I20260927 14:53:10.062326 11649 raft_consensus.cc:388] T 00000000000000000000000000000000 P 31f00e587ac6439aa5166ae2930cdef0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 14:53:10.062392 11649 raft_consensus.cc:482] T 00000000000000000000000000000000 P 31f00e587ac6439aa5166ae2930cdef0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 14:53:10.062490 11649 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 31f00e587ac6439aa5166ae2930cdef0 [term 0 FOLLOWER]: Advancing to term 1
I20260927 14:53:10.063586 11649 raft_consensus.cc:504] T 00000000000000000000000000000000 P 31f00e587ac6439aa5166ae2930cdef0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31f00e587ac6439aa5166ae2930cdef0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41989 } }
I20260927 14:53:10.063930 11649 leader_election.cc:302] T 00000000000000000000000000000000 P 31f00e587ac6439aa5166ae2930cdef0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 31f00e587ac6439aa5166ae2930cdef0; no voters:
I20260927 14:53:10.064291 11649 leader_election.cc:288] T 00000000000000000000000000000000 P 31f00e587ac6439aa5166ae2930cdef0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 14:53:10.064399 11654 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 31f00e587ac6439aa5166ae2930cdef0 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 14:53:10.065201 11654 raft_consensus.cc:686] T 00000000000000000000000000000000 P 31f00e587ac6439aa5166ae2930cdef0 [term 1 LEADER]: Becoming Leader. State: Replica: 31f00e587ac6439aa5166ae2930cdef0, State: Running, Role: LEADER
I20260927 14:53:10.065559 11654 consensus_queue.cc:237] T 00000000000000000000000000000000 P 31f00e587ac6439aa5166ae2930cdef0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31f00e587ac6439aa5166ae2930cdef0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41989 } }
I20260927 14:53:10.065635 11649 sys_catalog.cc:564] T 00000000000000000000000000000000 P 31f00e587ac6439aa5166ae2930cdef0 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 14:53:10.066673 11654 sys_catalog.cc:455] T 00000000000000000000000000000000 P 31f00e587ac6439aa5166ae2930cdef0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 31f00e587ac6439aa5166ae2930cdef0. Latest consensus state: current_term: 1 leader_uuid: "31f00e587ac6439aa5166ae2930cdef0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31f00e587ac6439aa5166ae2930cdef0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41989 } } }
I20260927 14:53:10.066875 11654 sys_catalog.cc:458] T 00000000000000000000000000000000 P 31f00e587ac6439aa5166ae2930cdef0 [sys.catalog]: This master's current role is: LEADER
I20260927 14:53:10.068069 11655 sys_catalog.cc:455] T 00000000000000000000000000000000 P 31f00e587ac6439aa5166ae2930cdef0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "31f00e587ac6439aa5166ae2930cdef0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31f00e587ac6439aa5166ae2930cdef0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41989 } } }
I20260927 14:53:10.068163 11655 sys_catalog.cc:458] T 00000000000000000000000000000000 P 31f00e587ac6439aa5166ae2930cdef0 [sys.catalog]: This master's current role is: LEADER
I20260927 14:53:10.068737 11649 ranger_client.cc:318] Using new properties file: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790520684602251-392-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260927 14:53:10.069957 11663 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 14:53:10.075925 11663 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 14:53:10.082530 11663 catalog_manager.cc:1347] Generated new cluster ID: e50e9cd65e6e44a59ed915cfce82cd6c
I20260927 14:53:10.082602 11663 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 14:53:10.100785 11663 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 14:53:10.102228 11663 catalog_manager.cc:1504] Loading token signing keys...
I20260927 14:53:10.120334 11663 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 31f00e587ac6439aa5166ae2930cdef0: Generated new TSK 0
I20260927 14:53:10.121322 11663 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790520684602251-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790520684602251-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260927 14:53:10.816813 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6936m5/build/debug/bin/kudu
/tmp/dist-test-task6936m5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790520684602251-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790520684602251-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790520684602251-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790520684602251-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:58651/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41989
--keytab_file=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790520684602251-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36119
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790520684602251-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790520684602251-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790520684602251-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 14:53:10.983496 11694 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 14:53:10.983896 11694 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 14:53:10.983975 11694 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 14:53:10.989956 11694 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 14:53:10.990048 11694 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 14:53:10.990150 11694 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 14:53:10.996032 11694 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36119
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790520684602251-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790520684602251-392-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.0.98.20:58651/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790520684602251-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790520684602251-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41989
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11694
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790520684602251-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:15:17 UTC on ccycloud.quasar-ledxfe.root.comops.site
build id 4676134
I20260927 14:53:10.997493 11694 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 14:53:10.998910 11694 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 14:53:11.011224 11704 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 14:53:11.012696 11702 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 14:53:11.013249 11701 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 14:53:11.014361 11694 server_base.cc:1034] running on GCE node
I20260927 14:53:11.014895 11694 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 14:53:11.015560 11694 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 14:53:11.018850 11694 hybrid_clock.cc:648] HybridClock initialized: now 1790520791018806 us; error 55 us; skew 500 ppm
Sep 27 14:53:11 dist-test-slave-xk1g krb5kdc[11137](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790520791, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 14:53:11.025352 11694 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 14:53:11.026355 11694 webserver.cc:460] Webserver started at http://127.0.98.1:36335/ using document root <none> and password file <none>
I20260927 14:53:11.026958 11694 fs_manager.cc:359] Metadata directory not provided
I20260927 14:53:11.027021 11694 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 14:53:11.027242 11694 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 27 14:53:11 dist-test-slave-xk1g krb5kdc[11137](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790520791, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260927 14:53:11.043946 11694 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790520684602251-392-0/minicluster-data/ts-0/data/instance:
uuid: "688443f70c9046ca83f7ae1afcc8bc7a"
format_stamp: "Formatted at 2026-09-27 14:53:11 on dist-test-slave-xk1g"
server_key: "320875420809fce2ae295f48d6b7ac7a"
server_key_iv: "40fa3548ec2ec4566af7fcd6979dc9bd"
server_key_version: "kuduclusterkey@0"
I20260927 14:53:11.046664 11694 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790520684602251-392-0/minicluster-data/ts-0/wal/instance:
uuid: "688443f70c9046ca83f7ae1afcc8bc7a"
format_stamp: "Formatted at 2026-09-27 14:53:11 on dist-test-slave-xk1g"
server_key: "320875420809fce2ae295f48d6b7ac7a"
server_key_iv: "40fa3548ec2ec4566af7fcd6979dc9bd"
server_key_version: "kuduclusterkey@0"
I20260927 14:53:11.058570 11694 fs_manager.cc:691] Time spent creating directory manager: real 0.011s user 0.003s sys 0.000s
I20260927 14:53:11.076992 11711 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 14:53:11.078946 11694 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.002s sys 0.000s
I20260927 14:53:11.079151 11694 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790520684602251-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790520684602251-392-0/minicluster-data/ts-0/wal
uuid: "688443f70c9046ca83f7ae1afcc8bc7a"
format_stamp: "Formatted at 2026-09-27 14:53:11 on dist-test-slave-xk1g"
server_key: "320875420809fce2ae295f48d6b7ac7a"
server_key_iv: "40fa3548ec2ec4566af7fcd6979dc9bd"
server_key_version: "kuduclusterkey@0"
I20260927 14:53:11.079373 11694 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790520684602251-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790520684602251-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790520684602251-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 14:53:11.119787 11694 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 14:53:11.123281 11694 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 14:53:11.123488 11694 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 14:53:11.124241 11694 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 14:53:11.125355 11694 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 14:53:11.125469 11694 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 14:53:11.125548 11694 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 14:53:11.125592 11694 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 14:53:11.137102 11694 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:46699
I20260927 14:53:11.138458 11694 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790520684602251-392-0/minicluster-data/ts-0/data/info.pb
I20260927 14:53:11.145236 11826 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:46699 every 8 connection(s)
I20260927 14:53:11.148778 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6936m5/build/debug/bin/kudu as pid 11694
I20260927 14:53:11.148890 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790520684602251-392-0/minicluster-data/ts-0/wal/instance
I20260927 14:53:11.149247 392 external_mini_cluster.cc:1428] Setting key 18225f682223d6c884037562fc9d8650
Sep 27 14:53:11 dist-test-slave-xk1g krb5kdc[11137](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790520791, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260927 14:53:11.167440 11829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:53:11.147604 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41989 (local address 127.0.98.1:52907)
0927 14:53:11.149983 (+ 2379us) negotiation.cc:107] Waiting for socket to connect
0927 14:53:11.150044 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0927 14:53:11.151057 (+ 1013us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:53:11.152749 (+ 1692us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:53:11.152760 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:53:11.153158 (+ 398us) client_negotiation.cc:189] Negotiated authn=SASL
0927 14:53:11.153888 (+ 730us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:53:11.153910 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:53:11.155176 (+ 1266us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:53:11.155180 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:53:11.156649 (+ 1469us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:53:11.156660 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:53:11.157453 (+ 793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:53:11.157458 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:53:11.157652 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:53:11.158353 (+ 701us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 14:53:11.158387 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0927 14:53:11.159977 (+ 1590us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 14:53:11.162425 (+ 2448us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 14:53:11.162433 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 14:53:11.162451 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0927 14:53:11.162856 (+ 405us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 14:53:11.164717 (+ 1861us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 14:53:11.164721 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 14:53:11.164723 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 14:53:11.164920 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 14:53:11.165468 (+ 548us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 14:53:11.165475 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 14:53:11.165815 (+ 340us) client_negotiation.cc:769] Sending connection context
0927 14:53:11.166040 (+ 225us) client_negotiation.cc:240] Negotiation successful
0927 14:53:11.166392 (+ 352us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1922,"thread_start_us":136,"threads_started":1}
I20260927 14:53:11.168176 11830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:53:11.149831 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:52907 (local address 127.0.98.62:41989)
0927 14:53:11.151885 (+ 2054us) server_negotiation.cc:206] Beginning negotiation
0927 14:53:11.151890 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 14:53:11.151906 (+ 16us) server_negotiation.cc:407] Connection header received
0927 14:53:11.151947 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:53:11.151950 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:53:11.151999 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:53:11.154044 (+ 2045us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:53:11.154157 (+ 113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:53:11.154993 (+ 836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:53:11.156791 (+ 1798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:53:11.157156 (+ 365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:53:11.157260 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:53:11.160188 (+ 2928us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:53:11.160192 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:53:11.160195 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 14:53:11.160220 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:53:11.162296 (+ 2076us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:53:11.163031 (+ 735us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:53:11.163035 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:53:11.163036 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:53:11.163131 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:53:11.165093 (+ 1962us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:53:11.165097 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:53:11.165098 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:53:11.165325 (+ 227us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:53:11.165415 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0927 14:53:11.167864 (+ 2449us) server_negotiation.cc:299] Negotiation successful
0927 14:53:11.168014 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1968,"thread_start_us":73,"threads_started":1}
I20260927 14:53:11.169982 11827 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41989
I20260927 14:53:11.170230 11827 heartbeater.cc:464] Registering TS with master...
I20260927 14:53:11.170785 11827 heartbeater.cc:511] Master 127.0.98.62:41989 requested a full tablet report, sending...
I20260927 14:53:11.172509 11613 ts_manager.cc:194] Registered new tserver with Master: 688443f70c9046ca83f7ae1afcc8bc7a (127.0.98.1:46699)
I20260927 14:53:11.174064 11613 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:52907
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790520684602251-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790520684602251-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260927 14:53:11.184942 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6936m5/build/debug/bin/kudu
/tmp/dist-test-task6936m5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790520684602251-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790520684602251-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790520684602251-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790520684602251-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:58651/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41989
--keytab_file=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790520684602251-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36119
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790520684602251-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790520684602251-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790520684602251-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 14:53:11 dist-test-slave-xk1g krb5kdc[11137](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790520791, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260927 14:53:11.333319 11833 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 14:53:11.333683 11833 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 14:53:11.333770 11833 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 14:53:11.338100 11833 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 14:53:11.338186 11833 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 14:53:11.338289 11833 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 14:53:11.343930 11833 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36119
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790520684602251-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790520684602251-392-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.0.98.20:58651/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790520684602251-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790520684602251-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41989
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11833
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790520684602251-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:15:17 UTC on ccycloud.quasar-ledxfe.root.comops.site
build id 4676134
I20260927 14:53:11.345046 11833 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 14:53:11.346122 11833 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 14:53:11.355919 11833 server_base.cc:1034] running on GCE node
W20260927 14:53:11.355831 11839 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 14:53:11.356302 11840 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 14:53:11.356426 11842 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 14:53:11.356989 11833 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 14:53:11.357846 11833 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 14:53:11.359035 11833 hybrid_clock.cc:648] HybridClock initialized: now 1790520791358964 us; error 87 us; skew 500 ppm
Sep 27 14:53:11 dist-test-slave-xk1g krb5kdc[11137](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790520791, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 14:53:11.362612 11833 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 14:53:11.363755 11833 webserver.cc:460] Webserver started at http://127.0.98.2:36219/ using document root <none> and password file <none>
I20260927 14:53:11.364562 11833 fs_manager.cc:359] Metadata directory not provided
I20260927 14:53:11.364720 11833 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 14:53:11.365020 11833 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 27 14:53:11 dist-test-slave-xk1g krb5kdc[11137](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790520791, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260927 14:53:11.385113 11833 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790520684602251-392-0/minicluster-data/ts-1/data/instance:
uuid: "e8e2177edc314f23a7b05d211924ae8d"
format_stamp: "Formatted at 2026-09-27 14:53:11 on dist-test-slave-xk1g"
server_key: "d02de86d000842bd07f6ee63a939fae7"
server_key_iv: "02cf44c519db48c955952eb334111591"
server_key_version: "kuduclusterkey@0"
I20260927 14:53:11.386011 11833 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790520684602251-392-0/minicluster-data/ts-1/wal/instance:
uuid: "e8e2177edc314f23a7b05d211924ae8d"
format_stamp: "Formatted at 2026-09-27 14:53:11 on dist-test-slave-xk1g"
server_key: "d02de86d000842bd07f6ee63a939fae7"
server_key_iv: "02cf44c519db48c955952eb334111591"
server_key_version: "kuduclusterkey@0"
I20260927 14:53:11.391307 11833 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260927 14:53:11.415203 11849 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 14:53:11.417128 11833 fs_manager.cc:725] Time spent opening block manager: real 0.011s user 0.004s sys 0.000s
I20260927 14:53:11.417404 11833 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790520684602251-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790520684602251-392-0/minicluster-data/ts-1/wal
uuid: "e8e2177edc314f23a7b05d211924ae8d"
format_stamp: "Formatted at 2026-09-27 14:53:11 on dist-test-slave-xk1g"
server_key: "d02de86d000842bd07f6ee63a939fae7"
server_key_iv: "02cf44c519db48c955952eb334111591"
server_key_version: "kuduclusterkey@0"
I20260927 14:53:11.417603 11833 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790520684602251-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790520684602251-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790520684602251-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 14:53:11.460489 11833 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 14:53:11.467155 11833 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 14:53:11.467433 11833 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 14:53:11.468210 11833 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 14:53:11.469376 11833 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 14:53:11.469499 11833 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 14:53:11.469602 11833 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 14:53:11.469673 11833 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 14:53:11.481244 11833 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:41687
I20260927 14:53:11.482546 11833 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790520684602251-392-0/minicluster-data/ts-1/data/info.pb
I20260927 14:53:11.486229 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6936m5/build/debug/bin/kudu as pid 11833
I20260927 14:53:11.486347 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790520684602251-392-0/minicluster-data/ts-1/wal/instance
I20260927 14:53:11.486675 392 external_mini_cluster.cc:1428] Setting key fa07c2472a2268972ddcc4498313d0cd
I20260927 14:53:11.492398 11962 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:41687 every 8 connection(s)
Sep 27 14:53:11 dist-test-slave-xk1g krb5kdc[11137](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790520791, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 14:53:11.514038 11830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:53:11.499585 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:52517 (local address 127.0.98.62:41989)
0927 14:53:11.499760 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0927 14:53:11.499765 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 14:53:11.501578 (+ 1813us) server_negotiation.cc:407] Connection header received
0927 14:53:11.502872 (+ 1294us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:53:11.502887 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:53:11.502943 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:53:11.502995 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:53:11.505129 (+ 2134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:53:11.505969 (+ 840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:53:11.507346 (+ 1377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:53:11.507582 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:53:11.507701 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:53:11.510350 (+ 2649us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:53:11.510353 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:53:11.510358 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 14:53:11.510389 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:53:11.511829 (+ 1440us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:53:11.512377 (+ 548us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:53:11.512381 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:53:11.512382 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:53:11.512462 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:53:11.512811 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:53:11.512814 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:53:11.512816 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:53:11.513024 (+ 208us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:53:11.513170 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0927 14:53:11.513711 (+ 541us) server_negotiation.cc:299] Negotiation successful
0927 14:53:11.513843 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260927 14:53:11.514935 11966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:53:11.498266 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41989 (local address 127.0.98.2:52517)
0927 14:53:11.501389 (+ 3123us) negotiation.cc:107] Waiting for socket to connect
0927 14:53:11.501451 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0927 14:53:11.502608 (+ 1157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:53:11.503410 (+ 802us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:53:11.503420 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:53:11.504241 (+ 821us) client_negotiation.cc:189] Negotiated authn=SASL
0927 14:53:11.504923 (+ 682us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:53:11.504941 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:53:11.506126 (+ 1185us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:53:11.506129 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:53:11.507209 (+ 1080us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:53:11.507218 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:53:11.507740 (+ 522us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:53:11.507744 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:53:11.507970 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:53:11.508642 (+ 672us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 14:53:11.508658 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 14:53:11.510184 (+ 1526us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 14:53:11.511990 (+ 1806us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 14:53:11.511996 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 14:53:11.512012 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0927 14:53:11.512268 (+ 256us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 14:53:11.512553 (+ 285us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 14:53:11.512557 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 14:53:11.512559 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 14:53:11.512714 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 14:53:11.513170 (+ 456us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 14:53:11.513177 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 14:53:11.513447 (+ 270us) client_negotiation.cc:769] Sending connection context
0927 14:53:11.513660 (+ 213us) client_negotiation.cc:240] Negotiation successful
0927 14:53:11.513991 (+ 331us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2884,"thread_start_us":196,"threads_started":1}
I20260927 14:53:11.516175 11963 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41989
I20260927 14:53:11.516431 11963 heartbeater.cc:464] Registering TS with master...
I20260927 14:53:11.516945 11963 heartbeater.cc:511] Master 127.0.98.62:41989 requested a full tablet report, sending...
I20260927 14:53:11.517982 11613 ts_manager.cc:194] Registered new tserver with Master: e8e2177edc314f23a7b05d211924ae8d (127.0.98.2:41687)
I20260927 14:53:11.518548 11613 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:52517
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790520684602251-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790520684602251-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260927 14:53:11.541430 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6936m5/build/debug/bin/kudu
/tmp/dist-test-task6936m5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790520684602251-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790520684602251-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790520684602251-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790520684602251-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:58651/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41989
--keytab_file=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790520684602251-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36119
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790520684602251-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790520684602251-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790520684602251-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 14:53:11.716919 11968 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 14:53:11.717742 11968 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 14:53:11.717875 11968 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 14:53:11.724349 11968 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 14:53:11.724505 11968 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 14:53:11.724647 11968 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 14:53:11.733254 11968 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36119
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790520684602251-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790520684602251-392-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.0.98.20:58651/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790520684602251-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790520684602251-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41989
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11968
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790520684602251-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:15:17 UTC on ccycloud.quasar-ledxfe.root.comops.site
build id 4676134
I20260927 14:53:11.734965 11968 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 14:53:11.736385 11968 file_cache.cc:493] Constructed file cache file cache with capacity 419430
Sep 27 14:53:11 dist-test-slave-xk1g krb5kdc[11137](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790520791, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
W20260927 14:53:11.777557 11974 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 14:53:11.777896 11976 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 14:53:11.778281 11968 server_base.cc:1034] running on GCE node
W20260927 14:53:11.789296 11973 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 14:53:11.790134 11968 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 14:53:11.791050 11968 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 14:53:11.792305 11968 hybrid_clock.cc:648] HybridClock initialized: now 1790520791792247 us; error 100 us; skew 500 ppm
Sep 27 14:53:11 dist-test-slave-xk1g krb5kdc[11137](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790520791, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 14:53:11.800469 11968 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 14:53:11.801841 11968 webserver.cc:460] Webserver started at http://127.0.98.3:35193/ using document root <none> and password file <none>
I20260927 14:53:11.803604 11968 fs_manager.cc:359] Metadata directory not provided
I20260927 14:53:11.804461 11968 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 14:53:11.804813 11968 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 27 14:53:11 dist-test-slave-xk1g krb5kdc[11137](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790520791, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260927 14:53:11.826198 11968 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790520684602251-392-0/minicluster-data/ts-2/data/instance:
uuid: "f5e817139a0745bcaee689a76f9a083c"
format_stamp: "Formatted at 2026-09-27 14:53:11 on dist-test-slave-xk1g"
server_key: "383ec2c02913dae3bbfebefa8d76c9d9"
server_key_iv: "452466903efc679f4abdc995d6ad1089"
server_key_version: "kuduclusterkey@0"
I20260927 14:53:11.827070 11968 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790520684602251-392-0/minicluster-data/ts-2/wal/instance:
uuid: "f5e817139a0745bcaee689a76f9a083c"
format_stamp: "Formatted at 2026-09-27 14:53:11 on dist-test-slave-xk1g"
server_key: "383ec2c02913dae3bbfebefa8d76c9d9"
server_key_iv: "452466903efc679f4abdc995d6ad1089"
server_key_version: "kuduclusterkey@0"
I20260927 14:53:11.840371 11968 fs_manager.cc:691] Time spent creating directory manager: real 0.013s user 0.004s sys 0.000s
I20260927 14:53:11.880650 11983 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 14:53:11.882616 11968 fs_manager.cc:725] Time spent opening block manager: real 0.015s user 0.003s sys 0.001s
I20260927 14:53:11.882768 11968 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790520684602251-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790520684602251-392-0/minicluster-data/ts-2/wal
uuid: "f5e817139a0745bcaee689a76f9a083c"
format_stamp: "Formatted at 2026-09-27 14:53:11 on dist-test-slave-xk1g"
server_key: "383ec2c02913dae3bbfebefa8d76c9d9"
server_key_iv: "452466903efc679f4abdc995d6ad1089"
server_key_version: "kuduclusterkey@0"
I20260927 14:53:11.882912 11968 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790520684602251-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790520684602251-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790520684602251-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 14:53:11.922088 11968 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 14:53:11.927954 11968 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 14:53:11.928198 11968 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 14:53:11.928877 11968 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 14:53:11.929975 11968 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 14:53:11.930079 11968 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 14:53:11.930178 11968 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 14:53:11.930299 11968 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 14:53:11.943228 11968 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:38475
I20260927 14:53:11.944502 11968 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790520684602251-392-0/minicluster-data/ts-2/data/info.pb
I20260927 14:53:11.946202 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6936m5/build/debug/bin/kudu as pid 11968
I20260927 14:53:11.946311 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790520684602251-392-0/minicluster-data/ts-2/wal/instance
I20260927 14:53:11.946623 392 external_mini_cluster.cc:1428] Setting key 1214e8ea0339f0c991d494d0a75ce3f3
I20260927 14:53:11.956413 12096 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:38475 every 8 connection(s)
Sep 27 14:53:11 dist-test-slave-xk1g krb5kdc[11137](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790520791, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 14:53:11.986914 11830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:53:11.956965 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:43195 (local address 127.0.98.62:41989)
0927 14:53:11.957123 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0927 14:53:11.957128 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 14:53:11.958830 (+ 1702us) server_negotiation.cc:407] Connection header received
0927 14:53:11.960085 (+ 1255us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:53:11.960088 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:53:11.960147 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:53:11.960201 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:53:11.961880 (+ 1679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:53:11.962688 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:53:11.964395 (+ 1707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:53:11.964736 (+ 341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:53:11.964807 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:53:11.982524 (+ 17717us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:53:11.982529 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:53:11.982532 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 14:53:11.982566 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:53:11.984355 (+ 1789us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:53:11.985013 (+ 658us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:53:11.985017 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:53:11.985019 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:53:11.985123 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:53:11.985561 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:53:11.985565 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:53:11.985566 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:53:11.985803 (+ 237us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:53:11.985896 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0927 14:53:11.986611 (+ 715us) server_negotiation.cc:299] Negotiation successful
0927 14:53:11.986747 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260927 14:53:11.988699 12099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:53:11.957423 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41989 (local address 127.0.98.3:43195)
0927 14:53:11.958649 (+ 1226us) negotiation.cc:107] Waiting for socket to connect
0927 14:53:11.958707 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0927 14:53:11.959848 (+ 1141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:53:11.960625 (+ 777us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:53:11.960636 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:53:11.961016 (+ 380us) client_negotiation.cc:189] Negotiated authn=SASL
0927 14:53:11.961698 (+ 682us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:53:11.961717 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:53:11.962858 (+ 1141us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:53:11.962861 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:53:11.964256 (+ 1395us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:53:11.964266 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:53:11.965036 (+ 770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:53:11.965039 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:53:11.965279 (+ 240us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:53:11.980536 (+ 15257us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 14:53:11.980562 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0927 14:53:11.982261 (+ 1699us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 14:53:11.984506 (+ 2245us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 14:53:11.984513 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 14:53:11.984533 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0927 14:53:11.984892 (+ 359us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 14:53:11.985230 (+ 338us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 14:53:11.985233 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 14:53:11.985235 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 14:53:11.985447 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 14:53:11.985964 (+ 517us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 14:53:11.985970 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 14:53:11.986320 (+ 350us) client_negotiation.cc:769] Sending connection context
0927 14:53:11.987158 (+ 838us) client_negotiation.cc:240] Negotiation successful
0927 14:53:11.987530 (+ 372us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":993,"thread_start_us":160,"threads_started":1}
I20260927 14:53:11.990137 12097 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41989
I20260927 14:53:11.990392 12097 heartbeater.cc:464] Registering TS with master...
I20260927 14:53:11.990979 12097 heartbeater.cc:511] Master 127.0.98.62:41989 requested a full tablet report, sending...
I20260927 14:53:11.992477 11613 ts_manager.cc:194] Registered new tserver with Master: f5e817139a0745bcaee689a76f9a083c (127.0.98.3:38475)
I20260927 14:53:11.993269 11613 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:43195
I20260927 14:53:11.996732 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 14:53:12.012130 11830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:53:12.001333 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32826 (local address 127.0.98.62:41989)
0927 14:53:12.001460 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0927 14:53:12.001464 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 14:53:12.002005 (+ 541us) server_negotiation.cc:407] Connection header received
0927 14:53:12.002203 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:53:12.002207 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:53:12.002256 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:53:12.002313 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:53:12.003420 (+ 1107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:53:12.004254 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:53:12.005554 (+ 1300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:53:12.005913 (+ 359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:53:12.005993 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:53:12.008001 (+ 2008us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:53:12.008006 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:53:12.008009 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 14:53:12.008041 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:53:12.010034 (+ 1993us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:53:12.010612 (+ 578us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:53:12.010616 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:53:12.010618 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:53:12.010716 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:53:12.011056 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:53:12.011060 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:53:12.011062 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:53:12.011288 (+ 226us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:53:12.011400 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0927 14:53:12.011801 (+ 401us) server_negotiation.cc:299] Negotiation successful
0927 14:53:12.011941 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260927 14:53:12.016342 11613 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:32826:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 14:53:12.021349 11613 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 14:53:12.054005 12109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:53:12.032320 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:41687 (local address 127.0.0.1:32886)
0927 14:53:12.037353 (+ 5033us) negotiation.cc:107] Waiting for socket to connect
0927 14:53:12.037414 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0927 14:53:12.037545 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:53:12.049124 (+ 11579us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:53:12.049128 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:53:12.049149 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 14:53:12.049257 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:53:12.049262 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:53:12.050739 (+ 1477us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:53:12.050743 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:53:12.051974 (+ 1231us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:53:12.051985 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:53:12.053400 (+ 1415us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:53:12.053404 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:53:12.053543 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:53:12.053586 (+ 43us) client_negotiation.cc:769] Sending connection context
0927 14:53:12.053702 (+ 116us) client_negotiation.cc:240] Negotiation successful
0927 14:53:12.053846 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4831,"thread_start_us":113,"threads_started":1}
I20260927 14:53:12.054646 12110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:53:12.033356 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32886 (local address 127.0.98.2:41687)
0927 14:53:12.037571 (+ 4215us) server_negotiation.cc:206] Beginning negotiation
0927 14:53:12.037578 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 14:53:12.037608 (+ 30us) server_negotiation.cc:407] Connection header received
0927 14:53:12.037695 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:53:12.037701 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:53:12.037936 (+ 235us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:53:12.038103 (+ 167us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 14:53:12.049425 (+ 11322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:53:12.050581 (+ 1156us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:53:12.052130 (+ 1549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:53:12.053016 (+ 886us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:53:12.053130 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:53:12.053301 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0927 14:53:12.054402 (+ 1101us) server_negotiation.cc:299] Negotiation successful
0927 14:53:12.054510 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4113,"thread_start_us":114,"threads_started":1}
I20260927 14:53:12.058202 11890 tablet_service.cc:1467] Processing CreateTablet for tablet bfef067f279e46f08371a275fcb11c58 (DEFAULT_TABLE table=test-table [id=52ab5c2d688a4759936be1ed04fe1a08]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 14:53:12.059520 11890 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bfef067f279e46f08371a275fcb11c58. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 14:53:12.062815 12112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:53:12.034909 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54166 (local address 127.0.98.1:46699)
0927 14:53:12.037354 (+ 2445us) server_negotiation.cc:206] Beginning negotiation
0927 14:53:12.037363 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 14:53:12.038606 (+ 1243us) server_negotiation.cc:407] Connection header received
0927 14:53:12.038695 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:53:12.038701 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:53:12.038908 (+ 207us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:53:12.039047 (+ 139us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 14:53:12.040626 (+ 1579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:53:12.041759 (+ 1133us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:53:12.059795 (+ 18036us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:53:12.060795 (+ 1000us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:53:12.060908 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:53:12.061100 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0927 14:53:12.062521 (+ 1421us) server_negotiation.cc:299] Negotiation successful
0927 14:53:12.062649 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2314,"thread_start_us":104,"threads_started":1}
I20260927 14:53:12.063345 12113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:53:12.034728 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:46699 (local address 127.0.0.1:54166)
0927 14:53:12.038381 (+ 3653us) negotiation.cc:107] Waiting for socket to connect
0927 14:53:12.038396 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 14:53:12.038495 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:53:12.040280 (+ 1785us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:53:12.040287 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:53:12.040313 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 14:53:12.040468 (+ 155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:53:12.040476 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:53:12.041931 (+ 1455us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:53:12.041935 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:53:12.046122 (+ 4187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:53:12.046133 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:53:12.061222 (+ 15089us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:53:12.061226 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:53:12.061351 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:53:12.061380 (+ 29us) client_negotiation.cc:769] Sending connection context
0927 14:53:12.063065 (+ 1685us) client_negotiation.cc:240] Negotiation successful
0927 14:53:12.063169 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3585,"thread_start_us":106,"threads_started":1}
I20260927 14:53:12.066597 11742 tablet_service.cc:1467] Processing CreateTablet for tablet bfef067f279e46f08371a275fcb11c58 (DEFAULT_TABLE table=test-table [id=52ab5c2d688a4759936be1ed04fe1a08]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 14:53:12.067766 11742 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bfef067f279e46f08371a275fcb11c58. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 14:53:12.078812 12116 tablet_bootstrap.cc:492] T bfef067f279e46f08371a275fcb11c58 P 688443f70c9046ca83f7ae1afcc8bc7a: Bootstrap starting.
I20260927 14:53:12.079677 12111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:53:12.035441 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:38475 (local address 127.0.0.1:45852)
0927 14:53:12.039293 (+ 3852us) negotiation.cc:107] Waiting for socket to connect
0927 14:53:12.039308 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 14:53:12.039392 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:53:12.045213 (+ 5821us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:53:12.045217 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:53:12.045237 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 14:53:12.045362 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:53:12.045370 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:53:12.047655 (+ 2285us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:53:12.047659 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:53:12.056302 (+ 8643us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:53:12.056314 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:53:12.079125 (+ 22811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:53:12.079130 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:53:12.079277 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:53:12.079309 (+ 32us) client_negotiation.cc:769] Sending connection context
0927 14:53:12.079391 (+ 82us) client_negotiation.cc:240] Negotiation successful
0927 14:53:12.079496 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3774,"thread_start_us":87,"threads_started":1}
I20260927 14:53:12.080880 12115 tablet_bootstrap.cc:492] T bfef067f279e46f08371a275fcb11c58 P e8e2177edc314f23a7b05d211924ae8d: Bootstrap starting.
I20260927 14:53:12.081723 12114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:53:12.035863 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45852 (local address 127.0.98.3:38475)
0927 14:53:12.044460 (+ 8597us) server_negotiation.cc:206] Beginning negotiation
0927 14:53:12.044474 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 14:53:12.044505 (+ 31us) server_negotiation.cc:407] Connection header received
0927 14:53:12.044594 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:53:12.044601 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:53:12.044847 (+ 246us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:53:12.045031 (+ 184us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 14:53:12.046295 (+ 1264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:53:12.047496 (+ 1201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:53:12.077759 (+ 30263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:53:12.078733 (+ 974us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:53:12.078872 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:53:12.079042 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0927 14:53:12.081433 (+ 2391us) server_negotiation.cc:299] Negotiation successful
0927 14:53:12.081557 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8461,"thread_start_us":117,"threads_started":1}
I20260927 14:53:12.083225 12115 tablet_bootstrap.cc:654] T bfef067f279e46f08371a275fcb11c58 P e8e2177edc314f23a7b05d211924ae8d: Neither blocks nor log segments found. Creating new log.
I20260927 14:53:12.084134 12115 log.cc:824] T bfef067f279e46f08371a275fcb11c58 P e8e2177edc314f23a7b05d211924ae8d: Log is configured to *not* fsync() on all Append() calls
I20260927 14:53:12.085278 12116 tablet_bootstrap.cc:654] T bfef067f279e46f08371a275fcb11c58 P 688443f70c9046ca83f7ae1afcc8bc7a: Neither blocks nor log segments found. Creating new log.
I20260927 14:53:12.084849 12023 tablet_service.cc:1467] Processing CreateTablet for tablet bfef067f279e46f08371a275fcb11c58 (DEFAULT_TABLE table=test-table [id=52ab5c2d688a4759936be1ed04fe1a08]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 14:53:12.085983 12023 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bfef067f279e46f08371a275fcb11c58. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 14:53:12.086125 12116 log.cc:824] T bfef067f279e46f08371a275fcb11c58 P 688443f70c9046ca83f7ae1afcc8bc7a: Log is configured to *not* fsync() on all Append() calls
I20260927 14:53:12.093801 12115 tablet_bootstrap.cc:492] T bfef067f279e46f08371a275fcb11c58 P e8e2177edc314f23a7b05d211924ae8d: No bootstrap required, opened a new log
I20260927 14:53:12.094039 12115 ts_tablet_manager.cc:1397] T bfef067f279e46f08371a275fcb11c58 P e8e2177edc314f23a7b05d211924ae8d: Time spent bootstrapping tablet: real 0.013s user 0.005s sys 0.000s
I20260927 14:53:12.099579 12115 raft_consensus.cc:348] T bfef067f279e46f08371a275fcb11c58 P e8e2177edc314f23a7b05d211924ae8d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8e2177edc314f23a7b05d211924ae8d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41687 } } peers { permanent_uuid: "688443f70c9046ca83f7ae1afcc8bc7a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46699 } } peers { permanent_uuid: "f5e817139a0745bcaee689a76f9a083c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38475 } }
I20260927 14:53:12.099910 12115 raft_consensus.cc:374] T bfef067f279e46f08371a275fcb11c58 P e8e2177edc314f23a7b05d211924ae8d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 14:53:12.099978 12115 raft_consensus.cc:729] T bfef067f279e46f08371a275fcb11c58 P e8e2177edc314f23a7b05d211924ae8d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e8e2177edc314f23a7b05d211924ae8d, State: Initialized, Role: FOLLOWER
I20260927 14:53:12.100517 12115 consensus_queue.cc:260] T bfef067f279e46f08371a275fcb11c58 P e8e2177edc314f23a7b05d211924ae8d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8e2177edc314f23a7b05d211924ae8d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41687 } } peers { permanent_uuid: "688443f70c9046ca83f7ae1afcc8bc7a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46699 } } peers { permanent_uuid: "f5e817139a0745bcaee689a76f9a083c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38475 } }
I20260927 14:53:12.101482 12119 tablet_bootstrap.cc:492] T bfef067f279e46f08371a275fcb11c58 P f5e817139a0745bcaee689a76f9a083c: Bootstrap starting.
I20260927 14:53:12.101718 12115 ts_tablet_manager.cc:1428] T bfef067f279e46f08371a275fcb11c58 P e8e2177edc314f23a7b05d211924ae8d: Time spent starting tablet: real 0.008s user 0.007s sys 0.000s
I20260927 14:53:12.102082 11963 heartbeater.cc:503] Master 127.0.98.62:41989 was elected leader, sending a full tablet report...
I20260927 14:53:12.102588 12116 tablet_bootstrap.cc:492] T bfef067f279e46f08371a275fcb11c58 P 688443f70c9046ca83f7ae1afcc8bc7a: No bootstrap required, opened a new log
I20260927 14:53:12.102804 12116 ts_tablet_manager.cc:1397] T bfef067f279e46f08371a275fcb11c58 P 688443f70c9046ca83f7ae1afcc8bc7a: Time spent bootstrapping tablet: real 0.024s user 0.004s sys 0.000s
I20260927 14:53:12.104730 12119 tablet_bootstrap.cc:654] T bfef067f279e46f08371a275fcb11c58 P f5e817139a0745bcaee689a76f9a083c: Neither blocks nor log segments found. Creating new log.
I20260927 14:53:12.105798 12119 log.cc:824] T bfef067f279e46f08371a275fcb11c58 P f5e817139a0745bcaee689a76f9a083c: Log is configured to *not* fsync() on all Append() calls
I20260927 14:53:12.107957 12116 raft_consensus.cc:348] T bfef067f279e46f08371a275fcb11c58 P 688443f70c9046ca83f7ae1afcc8bc7a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8e2177edc314f23a7b05d211924ae8d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41687 } } peers { permanent_uuid: "688443f70c9046ca83f7ae1afcc8bc7a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46699 } } peers { permanent_uuid: "f5e817139a0745bcaee689a76f9a083c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38475 } }
I20260927 14:53:12.108265 12116 raft_consensus.cc:374] T bfef067f279e46f08371a275fcb11c58 P 688443f70c9046ca83f7ae1afcc8bc7a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 14:53:12.108332 12116 raft_consensus.cc:729] T bfef067f279e46f08371a275fcb11c58 P 688443f70c9046ca83f7ae1afcc8bc7a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 688443f70c9046ca83f7ae1afcc8bc7a, State: Initialized, Role: FOLLOWER
I20260927 14:53:12.108919 12116 consensus_queue.cc:260] T bfef067f279e46f08371a275fcb11c58 P 688443f70c9046ca83f7ae1afcc8bc7a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8e2177edc314f23a7b05d211924ae8d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41687 } } peers { permanent_uuid: "688443f70c9046ca83f7ae1afcc8bc7a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46699 } } peers { permanent_uuid: "f5e817139a0745bcaee689a76f9a083c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38475 } }
I20260927 14:53:12.110106 12116 ts_tablet_manager.cc:1428] T bfef067f279e46f08371a275fcb11c58 P 688443f70c9046ca83f7ae1afcc8bc7a: Time spent starting tablet: real 0.007s user 0.005s sys 0.003s
I20260927 14:53:12.110373 12119 tablet_bootstrap.cc:492] T bfef067f279e46f08371a275fcb11c58 P f5e817139a0745bcaee689a76f9a083c: No bootstrap required, opened a new log
I20260927 14:53:12.110491 11827 heartbeater.cc:503] Master 127.0.98.62:41989 was elected leader, sending a full tablet report...
I20260927 14:53:12.110628 12119 ts_tablet_manager.cc:1397] T bfef067f279e46f08371a275fcb11c58 P f5e817139a0745bcaee689a76f9a083c: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.003s
I20260927 14:53:12.114904 12119 raft_consensus.cc:348] T bfef067f279e46f08371a275fcb11c58 P f5e817139a0745bcaee689a76f9a083c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8e2177edc314f23a7b05d211924ae8d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41687 } } peers { permanent_uuid: "688443f70c9046ca83f7ae1afcc8bc7a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46699 } } peers { permanent_uuid: "f5e817139a0745bcaee689a76f9a083c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38475 } }
I20260927 14:53:12.115207 12119 raft_consensus.cc:374] T bfef067f279e46f08371a275fcb11c58 P f5e817139a0745bcaee689a76f9a083c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 14:53:12.115262 12119 raft_consensus.cc:729] T bfef067f279e46f08371a275fcb11c58 P f5e817139a0745bcaee689a76f9a083c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f5e817139a0745bcaee689a76f9a083c, State: Initialized, Role: FOLLOWER
I20260927 14:53:12.115718 12119 consensus_queue.cc:260] T bfef067f279e46f08371a275fcb11c58 P f5e817139a0745bcaee689a76f9a083c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8e2177edc314f23a7b05d211924ae8d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41687 } } peers { permanent_uuid: "688443f70c9046ca83f7ae1afcc8bc7a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46699 } } peers { permanent_uuid: "f5e817139a0745bcaee689a76f9a083c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38475 } }
I20260927 14:53:12.116462 12097 heartbeater.cc:503] Master 127.0.98.62:41989 was elected leader, sending a full tablet report...
I20260927 14:53:12.116804 12119 ts_tablet_manager.cc:1428] T bfef067f279e46f08371a275fcb11c58 P f5e817139a0745bcaee689a76f9a083c: Time spent starting tablet: real 0.006s user 0.006s sys 0.001s
W20260927 14:53:12.152462 11828 tablet.cc:2369] T bfef067f279e46f08371a275fcb11c58 P 688443f70c9046ca83f7ae1afcc8bc7a: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 27 14:53:12 dist-test-slave-xk1g krb5kdc[11137](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790520791, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
W20260927 14:53:12.205322 12098 tablet.cc:2369] T bfef067f279e46f08371a275fcb11c58 P f5e817139a0745bcaee689a76f9a083c: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 14:53:12.246452 11964 tablet.cc:2369] T bfef067f279e46f08371a275fcb11c58 P e8e2177edc314f23a7b05d211924ae8d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 14:53:12.257403 12122 raft_consensus.cc:482] T bfef067f279e46f08371a275fcb11c58 P 688443f70c9046ca83f7ae1afcc8bc7a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 14:53:12.257654 12122 raft_consensus.cc:504] T bfef067f279e46f08371a275fcb11c58 P 688443f70c9046ca83f7ae1afcc8bc7a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8e2177edc314f23a7b05d211924ae8d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41687 } } peers { permanent_uuid: "688443f70c9046ca83f7ae1afcc8bc7a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46699 } } peers { permanent_uuid: "f5e817139a0745bcaee689a76f9a083c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38475 } }
I20260927 14:53:12.259120 12122 leader_election.cc:288] T bfef067f279e46f08371a275fcb11c58 P 688443f70c9046ca83f7ae1afcc8bc7a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e8e2177edc314f23a7b05d211924ae8d (127.0.98.2:41687), f5e817139a0745bcaee689a76f9a083c (127.0.98.3:38475)
I20260927 14:53:12.264819 12110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:53:12.260184 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:44109 (local address 127.0.98.2:41687)
0927 14:53:12.260461 (+ 277us) server_negotiation.cc:206] Beginning negotiation
0927 14:53:12.260467 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 14:53:12.260484 (+ 17us) server_negotiation.cc:407] Connection header received
0927 14:53:12.260547 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:53:12.260551 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:53:12.260603 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:53:12.260698 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 14:53:12.261036 (+ 338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:53:12.261841 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:53:12.263411 (+ 1570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:53:12.264068 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:53:12.264155 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:53:12.264227 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0927 14:53:12.264583 (+ 356us) server_negotiation.cc:299] Negotiation successful
0927 14:53:12.264676 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":199}
I20260927 14:53:12.265384 12127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:53:12.259342 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:41687 (local address 127.0.98.1:44109)
0927 14:53:12.260141 (+ 799us) negotiation.cc:107] Waiting for socket to connect
0927 14:53:12.260159 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 14:53:12.260279 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:53:12.260737 (+ 458us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:53:12.260740 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:53:12.260764 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 14:53:12.260877 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:53:12.260884 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:53:12.261995 (+ 1111us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:53:12.261998 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:53:12.263261 (+ 1263us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:53:12.263272 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:53:12.264327 (+ 1055us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:53:12.264331 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:53:12.264422 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:53:12.264446 (+ 24us) client_negotiation.cc:769] Sending connection context
0927 14:53:12.265096 (+ 650us) client_negotiation.cc:240] Negotiation successful
0927 14:53:12.265220 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":715,"thread_start_us":145,"threads_started":1}
I20260927 14:53:12.266134 12126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:53:12.259748 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:38475 (local address 127.0.98.1:33413)
0927 14:53:12.260966 (+ 1218us) negotiation.cc:107] Waiting for socket to connect
0927 14:53:12.260982 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 14:53:12.261155 (+ 173us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:53:12.261533 (+ 378us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:53:12.261536 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:53:12.261554 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 14:53:12.261662 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:53:12.261669 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:53:12.262743 (+ 1074us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:53:12.262746 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:53:12.264097 (+ 1351us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:53:12.264109 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:53:12.265621 (+ 1512us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:53:12.265628 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:53:12.265725 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:53:12.265752 (+ 27us) client_negotiation.cc:769] Sending connection context
0927 14:53:12.265902 (+ 150us) client_negotiation.cc:240] Negotiation successful
0927 14:53:12.265973 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1149,"mutex_wait_us":17,"thread_start_us":106,"threads_started":1}
I20260927 14:53:12.266175 12114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:53:12.260320 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:33413 (local address 127.0.98.3:38475)
0927 14:53:12.260440 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0927 14:53:12.260445 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 14:53:12.261086 (+ 641us) server_negotiation.cc:407] Connection header received
0927 14:53:12.261282 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:53:12.261286 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:53:12.261338 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:53:12.261391 (+ 53us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 14:53:12.261813 (+ 422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:53:12.262554 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:53:12.264255 (+ 1701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:53:12.264870 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:53:12.264963 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:53:12.265033 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0927 14:53:12.265948 (+ 915us) server_negotiation.cc:299] Negotiation successful
0927 14:53:12.266033 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260927 14:53:12.266104 11917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bfef067f279e46f08371a275fcb11c58" candidate_uuid: "688443f70c9046ca83f7ae1afcc8bc7a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e8e2177edc314f23a7b05d211924ae8d" is_pre_election: true
I20260927 14:53:12.266494 11917 raft_consensus.cc:2454] T bfef067f279e46f08371a275fcb11c58 P e8e2177edc314f23a7b05d211924ae8d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 688443f70c9046ca83f7ae1afcc8bc7a in term 0.
I20260927 14:53:12.267501 11712 leader_election.cc:302] T bfef067f279e46f08371a275fcb11c58 P 688443f70c9046ca83f7ae1afcc8bc7a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 688443f70c9046ca83f7ae1afcc8bc7a, e8e2177edc314f23a7b05d211924ae8d; no voters:
I20260927 14:53:12.267637 12051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bfef067f279e46f08371a275fcb11c58" candidate_uuid: "688443f70c9046ca83f7ae1afcc8bc7a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f5e817139a0745bcaee689a76f9a083c" is_pre_election: true
I20260927 14:53:12.267848 12122 raft_consensus.cc:2781] T bfef067f279e46f08371a275fcb11c58 P 688443f70c9046ca83f7ae1afcc8bc7a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 14:53:12.267948 12122 raft_consensus.cc:482] T bfef067f279e46f08371a275fcb11c58 P 688443f70c9046ca83f7ae1afcc8bc7a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 14:53:12.268007 12122 raft_consensus.cc:3037] T bfef067f279e46f08371a275fcb11c58 P 688443f70c9046ca83f7ae1afcc8bc7a [term 0 FOLLOWER]: Advancing to term 1
I20260927 14:53:12.267968 12051 raft_consensus.cc:2454] T bfef067f279e46f08371a275fcb11c58 P f5e817139a0745bcaee689a76f9a083c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 688443f70c9046ca83f7ae1afcc8bc7a in term 0.
I20260927 14:53:12.269402 12122 raft_consensus.cc:504] T bfef067f279e46f08371a275fcb11c58 P 688443f70c9046ca83f7ae1afcc8bc7a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8e2177edc314f23a7b05d211924ae8d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41687 } } peers { permanent_uuid: "688443f70c9046ca83f7ae1afcc8bc7a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46699 } } peers { permanent_uuid: "f5e817139a0745bcaee689a76f9a083c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38475 } }
I20260927 14:53:12.269798 12122 leader_election.cc:288] T bfef067f279e46f08371a275fcb11c58 P 688443f70c9046ca83f7ae1afcc8bc7a [CANDIDATE]: Term 1 election: Requested vote from peers e8e2177edc314f23a7b05d211924ae8d (127.0.98.2:41687), f5e817139a0745bcaee689a76f9a083c (127.0.98.3:38475)
I20260927 14:53:12.270179 12051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bfef067f279e46f08371a275fcb11c58" candidate_uuid: "688443f70c9046ca83f7ae1afcc8bc7a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f5e817139a0745bcaee689a76f9a083c"
I20260927 14:53:12.270330 12051 raft_consensus.cc:3037] T bfef067f279e46f08371a275fcb11c58 P f5e817139a0745bcaee689a76f9a083c [term 0 FOLLOWER]: Advancing to term 1
I20260927 14:53:12.271689 12051 raft_consensus.cc:2454] T bfef067f279e46f08371a275fcb11c58 P f5e817139a0745bcaee689a76f9a083c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 688443f70c9046ca83f7ae1afcc8bc7a in term 1.
I20260927 14:53:12.271961 11917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bfef067f279e46f08371a275fcb11c58" candidate_uuid: "688443f70c9046ca83f7ae1afcc8bc7a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e8e2177edc314f23a7b05d211924ae8d"
I20260927 14:53:12.272105 11917 raft_consensus.cc:3037] T bfef067f279e46f08371a275fcb11c58 P e8e2177edc314f23a7b05d211924ae8d [term 0 FOLLOWER]: Advancing to term 1
I20260927 14:53:12.273308 11715 leader_election.cc:302] T bfef067f279e46f08371a275fcb11c58 P 688443f70c9046ca83f7ae1afcc8bc7a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 688443f70c9046ca83f7ae1afcc8bc7a, f5e817139a0745bcaee689a76f9a083c; no voters:
I20260927 14:53:12.273502 11917 raft_consensus.cc:2454] T bfef067f279e46f08371a275fcb11c58 P e8e2177edc314f23a7b05d211924ae8d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 688443f70c9046ca83f7ae1afcc8bc7a in term 1.
I20260927 14:53:12.273579 12122 raft_consensus.cc:2781] T bfef067f279e46f08371a275fcb11c58 P 688443f70c9046ca83f7ae1afcc8bc7a [term 1 FOLLOWER]: Leader election won for term 1
I20260927 14:53:12.273811 12122 raft_consensus.cc:686] T bfef067f279e46f08371a275fcb11c58 P 688443f70c9046ca83f7ae1afcc8bc7a [term 1 LEADER]: Becoming Leader. State: Replica: 688443f70c9046ca83f7ae1afcc8bc7a, State: Running, Role: LEADER
I20260927 14:53:12.274214 12122 consensus_queue.cc:237] T bfef067f279e46f08371a275fcb11c58 P 688443f70c9046ca83f7ae1afcc8bc7a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8e2177edc314f23a7b05d211924ae8d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41687 } } peers { permanent_uuid: "688443f70c9046ca83f7ae1afcc8bc7a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46699 } } peers { permanent_uuid: "f5e817139a0745bcaee689a76f9a083c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38475 } }
I20260927 14:53:12.277949 11613 catalog_manager.cc:5495] T bfef067f279e46f08371a275fcb11c58 P 688443f70c9046ca83f7ae1afcc8bc7a reported cstate change: term changed from 0 to 1, leader changed from <none> to 688443f70c9046ca83f7ae1afcc8bc7a (127.0.98.1). New cstate: current_term: 1 leader_uuid: "688443f70c9046ca83f7ae1afcc8bc7a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8e2177edc314f23a7b05d211924ae8d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41687 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "688443f70c9046ca83f7ae1afcc8bc7a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46699 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f5e817139a0745bcaee689a76f9a083c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38475 } health_report { overall_health: UNKNOWN } } }
I20260927 14:53:12.337852 12112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:53:12.332286 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54178 (local address 127.0.98.1:46699)
0927 14:53:12.332422 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0927 14:53:12.332427 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 14:53:12.332445 (+ 18us) server_negotiation.cc:407] Connection header received
0927 14:53:12.332670 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:53:12.332674 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:53:12.332754 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:53:12.332865 (+ 111us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 14:53:12.333274 (+ 409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:53:12.334101 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:53:12.335462 (+ 1361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:53:12.335833 (+ 371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:53:12.335947 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:53:12.336405 (+ 458us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 14:53:12.336964 (+ 559us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 14:53:12.337122 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0927 14:53:12.337599 (+ 477us) server_negotiation.cc:299] Negotiation successful
0927 14:53:12.337668 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260927 14:53:12.345830 12051 raft_consensus.cc:1261] T bfef067f279e46f08371a275fcb11c58 P f5e817139a0745bcaee689a76f9a083c [term 1 FOLLOWER]: Refusing update from remote peer 688443f70c9046ca83f7ae1afcc8bc7a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 14:53:12.346953 12122 consensus_queue.cc:1035] T bfef067f279e46f08371a275fcb11c58 P 688443f70c9046ca83f7ae1afcc8bc7a [LEADER]: Connected to new peer: Peer: permanent_uuid: "f5e817139a0745bcaee689a76f9a083c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38475 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 14:53:12.350849 11917 raft_consensus.cc:1261] T bfef067f279e46f08371a275fcb11c58 P e8e2177edc314f23a7b05d211924ae8d [term 1 FOLLOWER]: Refusing update from remote peer 688443f70c9046ca83f7ae1afcc8bc7a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 14:53:12.351528 12122 consensus_queue.cc:1035] T bfef067f279e46f08371a275fcb11c58 P 688443f70c9046ca83f7ae1afcc8bc7a [LEADER]: Connected to new peer: Peer: permanent_uuid: "e8e2177edc314f23a7b05d211924ae8d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41687 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 14:53:12.357403 12131 mvcc.cc:205] Tried to move back new op lower bound from 7333973165439340544 to 7333973165158580224. Current Snapshot: MvccSnapshot[applied={T|T < 7333973165439340544}]
I20260927 14:53:12.362056 12132 mvcc.cc:205] Tried to move back new op lower bound from 7333973165439340544 to 7333973165158580224. Current Snapshot: MvccSnapshot[applied={T|T < 7333973165439340544}]
I20260927 14:53:12.368412 11613 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:32826:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 14:53:12.368602 11613 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:32826:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 14:53:12.372442 11613 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 31f00e587ac6439aa5166ae2930cdef0: Sending DeleteTablet for 3 replicas of tablet bfef067f279e46f08371a275fcb11c58
I20260927 14:53:12.375330 11890 tablet_service.cc:1514] Processing DeleteTablet for tablet bfef067f279e46f08371a275fcb11c58 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 14:53:12 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:32886
I20260927 14:53:12.376044 11742 tablet_service.cc:1514] Processing DeleteTablet for tablet bfef067f279e46f08371a275fcb11c58 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 14:53:12 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:54166
I20260927 14:53:12.376596 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6936m5/build/debug/bin/kudu with pid 11694
I20260927 14:53:12.376716 12023 tablet_service.cc:1514] Processing DeleteTablet for tablet bfef067f279e46f08371a275fcb11c58 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 14:53:12 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:45852
I20260927 14:53:12.381320 12140 tablet_replica.cc:330] T bfef067f279e46f08371a275fcb11c58 P e8e2177edc314f23a7b05d211924ae8d: stopping tablet replica
I20260927 14:53:12.381672 12140 raft_consensus.cc:2229] T bfef067f279e46f08371a275fcb11c58 P e8e2177edc314f23a7b05d211924ae8d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 14:53:12.381976 12142 tablet_replica.cc:330] T bfef067f279e46f08371a275fcb11c58 P f5e817139a0745bcaee689a76f9a083c: stopping tablet replica
I20260927 14:53:12.382012 12140 raft_consensus.cc:2258] T bfef067f279e46f08371a275fcb11c58 P e8e2177edc314f23a7b05d211924ae8d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 14:53:12.382378 12142 raft_consensus.cc:2229] T bfef067f279e46f08371a275fcb11c58 P f5e817139a0745bcaee689a76f9a083c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 14:53:12.383599 12140 ts_tablet_manager.cc:1905] T bfef067f279e46f08371a275fcb11c58 P e8e2177edc314f23a7b05d211924ae8d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 14:53:12.384578 12142 raft_consensus.cc:2258] T bfef067f279e46f08371a275fcb11c58 P f5e817139a0745bcaee689a76f9a083c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 14:53:12.386341 12142 ts_tablet_manager.cc:1905] T bfef067f279e46f08371a275fcb11c58 P f5e817139a0745bcaee689a76f9a083c: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260927 14:53:12.387744 11851 connection.cc:535] server connection from 127.0.98.1:44109 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260927 14:53:12.388370 11600 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:46699 (error 108)
I20260927 14:53:12.388515 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6936m5/build/debug/bin/kudu with pid 11833
I20260927 14:53:12.390501 12111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:53:12.389528 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:46699 (local address 127.0.0.1:54182)
0927 14:53:12.390300 (+ 772us) negotiation.cc:107] Waiting for socket to connect
0927 14:53:12.390390 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:46699: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":646}
W20260927 14:53:12.390767 11600 catalog_manager.cc:4584] TS 688443f70c9046ca83f7ae1afcc8bc7a (127.0.98.1:46699): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet bfef067f279e46f08371a275fcb11c58: Network error: Client connection negotiation failed: client connection to 127.0.98.1:46699: connect: Connection refused (error 111)
I20260927 14:53:12.392192 12142 ts_tablet_manager.cc:1918] T bfef067f279e46f08371a275fcb11c58 P f5e817139a0745bcaee689a76f9a083c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 14:53:12.392318 12142 log.cc:1196] T bfef067f279e46f08371a275fcb11c58 P f5e817139a0745bcaee689a76f9a083c: Deleting WAL directory at /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790520684602251-392-0/minicluster-data/ts-2/wal/wals/bfef067f279e46f08371a275fcb11c58
I20260927 14:53:12.392670 12142 ts_tablet_manager.cc:1939] T bfef067f279e46f08371a275fcb11c58 P f5e817139a0745bcaee689a76f9a083c: Deleting consensus metadata
I20260927 14:53:12.394172 11600 catalog_manager.cc:4841] TS f5e817139a0745bcaee689a76f9a083c (127.0.98.3:38475): tablet bfef067f279e46f08371a275fcb11c58 (table test-table [id=52ab5c2d688a4759936be1ed04fe1a08]) successfully deleted
I20260927 14:53:12.396340 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6936m5/build/debug/bin/kudu with pid 11968
I20260927 14:53:12.400511 12111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:53:12.396531 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:41687 (local address 127.0.0.1:32894)
0927 14:53:12.400327 (+ 3796us) negotiation.cc:107] Waiting for socket to connect
0927 14:53:12.400403 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:41687: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":3674}
I20260927 14:53:12.404500 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6936m5/build/debug/bin/kudu with pid 11581
I20260927 14:53:12.409983 392 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260927 14:53:12.754456 392 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260927 14:53:12.754621 392 mini_ranger.cc:66] Stopping Ranger...
I20260927 14:53:13.099470 392 mini_ranger.cc:68] Stopped Ranger
2026-09-27 14:53:13.099 UTC [11167] LOG: received smart shutdown request
2026-09-27 14:53:13.102 UTC [11167] LOG: background worker "logical replication launcher" (PID 11176) exited with exit code 1
2026-09-27 14:53:13.103 UTC [11171] LOG: shutting down
2026-09-27 14:53:13.271 UTC [11167] LOG: database system is shut down
2026-09-27T14:53:13Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (46903 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790520684602251-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 14:53:13 dist-test-slave-xk1g krb5kdc[12152](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 14:53:13 dist-test-slave-xk1g krb5kdc[12152](info): set up 2 sockets
Sep 27 14:53:13 dist-test-slave-xk1g krb5kdc[12152](info): commencing operation
krb5kdc: starting...
W20260927 14:53:14.425165 392 mini_kdc.cc:121] Time spent starting KDC: real 1.021s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790520684602251-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790520684602251-392-0/krb5kdc/test-user.keytab.
Sep 27 14:53:14 dist-test-slave-xk1g krb5kdc[12152](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790520794, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T14:53:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T14:53:14Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790520684602251-392-0/krb5kdc/rangeradmin_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790520684602251-392-0/krb5kdc/rangeradmin_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790520684602251-392-0/krb5kdc/rangerlookup_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790520684602251-392-0/krb5kdc/rangerlookup_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790520684602251-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790520684602251-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
I20260927 14:53:14.589704 392 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790520684602251-392-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-task6936m5/build/debug/bin/postgres/pg_ctl -D /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790520684602251-392-0/minicluster-data/postgres -l logfile start
2026-09-27 14:53:17.035 UTC [12182] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-27 14:53:17.035 UTC [12182] LOG: listening on IPv4 address "127.0.98.20", port 43915
2026-09-27 14:53:17.039 UTC [12182] LOG: listening on Unix socket "/tmp/.s.PGSQL.43915"
2026-09-27 14:53:17.060 UTC [12185] LOG: database system was shut down at 2026-09-27 14:53:15 UTC
2026-09-27 14:53:17.063 UTC [12182] LOG: database system is ready to accept connections
I20260927 14:53:18.009037 392 mini_postgres.cc:96] Postgres bound to 43915
2026-09-27 14:53:18.016 UTC [12193] FATAL: database "slave" does not exist
127.0.98.20:43915 - accepting connections
I20260927 14:53:18.016525 392 mini_ranger.cc:161] Starting Ranger...
I20260927 14:53:18.031386 392 mini_ranger.cc:84] Created miniranger Postgres user
I20260927 14:53:18.509395 392 mini_ranger.cc:87] Created ranger Postgres database
I20260927 14:53:18.509486 392 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790520684602251-392-0/minicluster-data/ranger-admin
2026-09-27 14:53:18,779 [I] DB FLAVOR :POSTGRES
2026-09-27 14:53:18,779 [I] --------- Verifying Ranger DB connection ---------
2026-09-27 14:53:18,780 [I] Checking connection..
2026-09-27 14:53:18,780 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6936m5/build/debug/bin/postgresql.jar:/tmp/dist-test-task6936m5/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:43915/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-27 14:53:19,043 [I] Checking connection passed.
2026-09-27 14:53:19,043 [I] --------- Verifying version history table ---------
2026-09-27 14:53:19,043 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6936m5/build/debug/bin/postgresql.jar:/tmp/dist-test-task6936m5/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:43915/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-27 14:53:19,269 [I] Table x_db_version_h does not exist in database ranger
2026-09-27 14:53:19,270 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6936m5/build/debug/bin/postgresql.jar:/tmp/dist-test-task6936m5/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:43915/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-27 14:53:19,501 [I] Table x_db_version_h does not exist in database ranger
2026-09-27 14:53:19,502 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-27 14:53:19,502 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6936m5/build/debug/bin/postgresql.jar:/tmp/dist-test-task6936m5/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:43915/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task6936m5/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-27 14:53:19.733 UTC [12257] WARNING: there is no transaction in progress
2026-09-27 14:53:19,739 [I] create_dbversion_catalog.sql file imported successfully
2026-09-27 14:53:19,739 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6936m5/build/debug/bin/postgresql.jar:/tmp/dist-test-task6936m5/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:43915/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-27 14:53:19,997 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-27 14:53:19,997 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-27 14:53:19,998 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6936m5/build/debug/bin/postgresql.jar:/tmp/dist-test-task6936m5/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:43915/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-27 14:53:20,233 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6936m5/build/debug/bin/postgresql.jar:/tmp/dist-test-task6936m5/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:43915/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-27 14:53:20,478 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6936m5/build/debug/bin/postgresql.jar:/tmp/dist-test-task6936m5/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:43915/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-xk1g.c.gcp-upstream.internal','N') ;"
2026-09-27 14:53:20,713 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6936m5/build/debug/bin/postgresql.jar:/tmp/dist-test-task6936m5/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:43915/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-27 14:53:20,961 [I] Table x_portal_user does not exist in database ranger
2026-09-27 14:53:20,961 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6936m5/build/debug/bin/postgresql.jar:/tmp/dist-test-task6936m5/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:43915/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-27 14:53:21,209 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-27 14:53:21,209 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6936m5/build/debug/bin/postgresql.jar:/tmp/dist-test-task6936m5/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:43915/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-27 14:53:21,463 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6936m5/build/debug/bin/postgresql.jar:/tmp/dist-test-task6936m5/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:43915/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-27 14:53:21,685 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-27 14:53:21,686 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6936m5/build/debug/bin/postgresql.jar:/tmp/dist-test-task6936m5/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:43915/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task6936m5/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-27 14:53:22.674 UTC [12365] WARNING: there is no transaction in progress
2026-09-27 14:53:22.690 UTC [12365] WARNING: there is no transaction in progress
2026-09-27 14:53:22.705 UTC [12365] WARNING: there is no transaction in progress
2026-09-27 14:53:22.721 UTC [12365] WARNING: there is no transaction in progress
2026-09-27 14:53:22.740 UTC [12365] WARNING: there is no transaction in progress
2026-09-27 14:53:22.758 UTC [12365] WARNING: there is no transaction in progress
2026-09-27 14:53:22.815 UTC [12365] WARNING: there is no transaction in progress
2026-09-27 14:53:22.821 UTC [12365] WARNING: there is no transaction in progress
2026-09-27 14:53:22.836 UTC [12365] WARNING: there is no transaction in progress
2026-09-27 14:53:22.846 UTC [12365] WARNING: there is no transaction in progress
2026-09-27 14:53:22.855 UTC [12365] WARNING: there is no transaction in progress
2026-09-27 14:53:22.869 UTC [12365] WARNING: there is no transaction in progress
2026-09-27 14:53:22.879 UTC [12365] WARNING: there is no transaction in progress
2026-09-27 14:53:22.885 UTC [12365] WARNING: there is no transaction in progress
2026-09-27 14:53:22.892 UTC [12365] WARNING: there is no transaction in progress
2026-09-27 14:53:22.898 UTC [12365] WARNING: there is no transaction in progress
2026-09-27 14:53:23,523 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-27 14:53:23,523 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6936m5/build/debug/bin/postgresql.jar:/tmp/dist-test-task6936m5/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:43915/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-27 14:53:23,747 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-27 14:53:23,747 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6936m5/build/debug/bin/postgresql.jar:/tmp/dist-test-task6936m5/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:43915/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-27 14:53:23,988 [I] Table x_portal_user already exists in database 'ranger'
2026-09-27 14:53:23,988 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6936m5/build/debug/bin/postgresql.jar:/tmp/dist-test-task6936m5/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:43915/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-27 14:53:24,231 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-27 14:53:24,232 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6936m5/build/debug/bin/postgresql.jar:/tmp/dist-test-task6936m5/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:43915/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-27 14:53:24,477 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6936m5/build/debug/bin/postgresql.jar:/tmp/dist-test-task6936m5/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:43915/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-27 14:53:24,731 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6936m5/build/debug/bin/postgresql.jar:/tmp/dist-test-task6936m5/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:43915/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-xk1g.c.gcp-upstream.internal';"
2026-09-27 14:53:24,954 [I] CORE_DB_SCHEMA import status has been updated
2026-09-27 14:53:24,954 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6936m5/build/debug/bin/postgresql.jar:/tmp/dist-test-task6936m5/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:43915/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-27 14:53:25,212 [I] DB_PATCHES have already been applied
I20260927 14:53:25.217725 392 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790520684602251-392-0/minicluster-data/ranger-admin:/tmp/dist-test-task6936m5/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-task6936m5/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-task6936m5/build/debug/bin/postgresql.jar:/tmp/dist-test-task6936m5/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260927 14:53:25.217789 392 mini_ranger.cc:193] Using host: 127.0.98.20
I20260927 14:53:25.221050 392 mini_ranger.cc:227] Ranger admin URL: http://127.0.98.20:50453
Sep 27, 2026 2:53:25 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 27, 2026 2:53:25 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 27, 2026 2:53:25 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790520684602251-392-0/minicluster-data/ranger-admin/ews/webapp
Sep 27, 2026 2:53:25 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790520684602251-392-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 27, 2026 2:53:25 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790520684602251-392-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 27, 2026 2:53:26 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790520684602251-392-0/minicluster-data/ranger-admin/ews/webapp].
Sep 27, 2026 2:53:26 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260927 14:53:26.323766 392 mini_ranger.cc:160] Time spent starting Ranger: real 8.307s user 0.000s sys 0.010s
Sep 27, 2026 2:53:45 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 27, 2026 2:53:45 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Sep 27, 2026 2:53:45 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 27, 2026 2:53:45 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 27, 2026 2:53:45 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 27, 2026 2:53:45 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 27, 2026 2:53:45 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 27, 2026 2:53:45 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 27, 2026 2:53:45 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 27, 2026 2:53:45 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 27, 2026 2:53:45 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 27, 2026 2:53:45 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 27, 2026 2:53:45 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 27, 2026 2:53:45 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 27, 2026 2:53:45 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 27, 2026 2:53:45 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 27, 2026 2:53:45 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 27, 2026 2:53:46 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260927 14:53:46.969696 392 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790520684602251-392-0/krb5kdc/rangerkms_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790520684602251-392-0/krb5kdc/rangerkms_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.98.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.0.98.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790520684602251-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790520684602251-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 27 14:53:47 dist-test-slave-xk1g krb5kdc[12152](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790520827, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260927 14:53:47.058360 392 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260927 14:53:47.068157 392 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260927 14:53:47.945668 392 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260927 14:53:47.945763 392 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790520684602251-392-0/minicluster-data/ranger-kms
I20260927 14:53:47.945784 392 mini_ranger_kms.cc:227] Using postgres at 127.0.98.20:43915
2026-09-27 14:53:48,111 [I] DB FLAVOR :POSTGRES
2026-09-27 14:53:48,112 [I] --------- Verifying Ranger DB connection ---------
2026-09-27 14:53:48,112 [I] Checking connection
2026-09-27 14:53:48,112 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6936m5/build/debug/bin/postgresql.jar:/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790520684602251-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:43915/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-27 14:53:48,355 [I] connection success
2026-09-27 14:53:48,355 [I] --------- Verifying Ranger DB tables ---------
2026-09-27 14:53:48,356 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-27 14:53:48,356 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6936m5/build/debug/bin/postgresql.jar:/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790520684602251-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:43915/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-27 14:53:48,582 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-27 14:53:48,583 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-27 14:53:48,583 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-27 14:53:48,583 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6936m5/build/debug/bin/postgresql.jar:/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790520684602251-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:43915/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790520684602251-392-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-27 14:53:48,837 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260927 14:53:49.289943 392 mini_ranger_kms.cc:310] Created kms service
I20260927 14:53:49.467468 392 mini_ranger_kms.cc:326] Created kudu user
I20260927 14:53:49.514335 392 mini_ranger_kms.cc:343] Created rangerkms user
Sep 27 14:53:49 dist-test-slave-xk1g krb5kdc[12152](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790520827, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260927 14:53:49.872664 392 mini_ranger_kms.cc:384] Added ranger policy
I20260927 14:53:49.872808 392 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790520684602251-392-0/minicluster-data/ranger-kms:/tmp/dist-test-task6936m5/build/debug/bin/postgresql.jar:/tmp/dist-test-task6936m5/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-task6936m5/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260927 14:53:49.872828 392 mini_ranger_kms.cc:240] Using host: 127.0.98.20
I20260927 14:53:49.876483 392 mini_ranger_kms.cc:276] Ranger KMS PID: 12545
I20260927 14:53:49.876561 392 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.0.98.20:37591
Sep 27, 2026 2:53:51 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 27, 2026 2:53:51 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 27, 2026 2:53:51 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790520684602251-392-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 27, 2026 2:53:51 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790520684602251-392-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 27, 2026 2:53:51 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790520684602251-392-0/minicluster-data/ranger-kms/ews/webapp].
Sep 27, 2026 2:53:51 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260927 14:53:52.111826 392 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.053s user 0.001s sys 0.150s
I20260927 14:53:52.111943 392 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260927 14:53:52.112010 392 mini_ranger_kms.cc:401] 127.0.98.20:37591/kms/v1/keys
Sep 27 14:53:52 dist-test-slave-xk1g krb5kdc[12152](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790520832, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 27 14:53:52 dist-test-slave-xk1g krb5kdc[12152](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790520832, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
Sep 27 14:53:53 dist-test-slave-xk1g krb5kdc[12152](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790520832, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
Sep 27 14:53:55 dist-test-slave-xk1g krb5kdc[12152](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790520835, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790520684602251-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790520684602251-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 14:53:55.474185 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6936m5/build/debug/bin/kudu
/tmp/dist-test-task6936m5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790520684602251-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790520684602251-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790520684602251-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790520684602251-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42115
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790520684602251-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33451
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:42115
--ranger_config_path=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790520684602251-392-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:37591/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790520684602251-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790520684602251-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790520684602251-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 14:53:55.581439 12596 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 14:53:55.581681 12596 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 14:53:55.581718 12596 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 14:53:55.585693 12596 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 14:53:55.585740 12596 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 14:53:55.585759 12596 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 14:53:55.585775 12596 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 14:53:55.585837 12596 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 14:53:55.591300 12596 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33451
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790520684602251-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790520684602251-392-0/minicluster-data/master-0/wal
--ranger_kms_url=127.0.98.20:37591/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:42115
--ranger_config_path=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790520684602251-392-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790520684602251-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42115
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790520684602251-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12596
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790520684602251-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:15:17 UTC on ccycloud.quasar-ledxfe.root.comops.site
build id 4676134
I20260927 14:53:55.592607 12596 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 14:53:55.593580 12596 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 14:53:55.601462 12604 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 14:53:55.601456 12601 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 14:53:55.601609 12602 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 14:53:55.601722 12596 server_base.cc:1034] running on GCE node
I20260927 14:53:55.602351 12596 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 14:53:55.603353 12596 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 14:53:55.604537 12596 hybrid_clock.cc:648] HybridClock initialized: now 1790520835604511 us; error 37 us; skew 500 ppm
Sep 27 14:53:55 dist-test-slave-xk1g krb5kdc[12152](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790520835, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 14:53:55.612993 12596 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 14:53:55.613936 12596 webserver.cc:460] Webserver started at http://127.0.98.62:46313/ using document root <none> and password file <none>
I20260927 14:53:55.614467 12596 fs_manager.cc:359] Metadata directory not provided
I20260927 14:53:55.614547 12596 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 14:53:55.614768 12596 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 27 14:53:55 dist-test-slave-xk1g krb5kdc[12152](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790520835, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260927 14:53:55.676388 12596 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790520684602251-392-0/minicluster-data/master-0/data/instance:
uuid: "ae16374104dc4823a1362be92f41ed52"
format_stamp: "Formatted at 2026-09-27 14:53:55 on dist-test-slave-xk1g"
server_key: "b417a96899809b6696e1d87e4609327d"
server_key_iv: "690a343ff21ebd64374fba4001984df6"
server_key_version: "kuduclusterkey@0"
I20260927 14:53:55.676987 12596 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790520684602251-392-0/minicluster-data/master-0/wal/instance:
uuid: "ae16374104dc4823a1362be92f41ed52"
format_stamp: "Formatted at 2026-09-27 14:53:55 on dist-test-slave-xk1g"
server_key: "b417a96899809b6696e1d87e4609327d"
server_key_iv: "690a343ff21ebd64374fba4001984df6"
server_key_version: "kuduclusterkey@0"
I20260927 14:53:55.680889 12596 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.001s
I20260927 14:53:55.698329 12611 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 14:53:55.699770 12596 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260927 14:53:55.699883 12596 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790520684602251-392-0/minicluster-data/master-0/data,/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790520684602251-392-0/minicluster-data/master-0/wal
uuid: "ae16374104dc4823a1362be92f41ed52"
format_stamp: "Formatted at 2026-09-27 14:53:55 on dist-test-slave-xk1g"
server_key: "b417a96899809b6696e1d87e4609327d"
server_key_iv: "690a343ff21ebd64374fba4001984df6"
server_key_version: "kuduclusterkey@0"
I20260927 14:53:55.700039 12596 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790520684602251-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790520684602251-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790520684602251-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 14:53:55.715958 12596 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 14:53:55.718796 12596 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 14:53:55.718986 12596 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 14:53:55.727000 12596 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:42115
I20260927 14:53:55.727005 12663 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:42115 every 8 connection(s)
I20260927 14:53:55.727928 12596 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790520684602251-392-0/minicluster-data/master-0/data/info.pb
I20260927 14:53:55.730654 12664 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 14:53:55.731335 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6936m5/build/debug/bin/kudu as pid 12596
I20260927 14:53:55.731453 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790520684602251-392-0/minicluster-data/master-0/wal/instance
I20260927 14:53:55.731753 392 external_mini_cluster.cc:1428] Setting key 9e3d8342b3aab14cbccbf2546c231857
I20260927 14:53:55.736907 12664 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ae16374104dc4823a1362be92f41ed52: Bootstrap starting.
I20260927 14:53:55.739326 12664 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ae16374104dc4823a1362be92f41ed52: Neither blocks nor log segments found. Creating new log.
I20260927 14:53:55.740187 12664 log.cc:824] T 00000000000000000000000000000000 P ae16374104dc4823a1362be92f41ed52: Log is configured to *not* fsync() on all Append() calls
Sep 27 14:53:55 dist-test-slave-xk1g krb5kdc[12152](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790520835, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 14:53:55.742328 12664 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ae16374104dc4823a1362be92f41ed52: No bootstrap required, opened a new log
I20260927 14:53:55.746621 12664 raft_consensus.cc:348] T 00000000000000000000000000000000 P ae16374104dc4823a1362be92f41ed52 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae16374104dc4823a1362be92f41ed52" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42115 } }
I20260927 14:53:55.746898 12664 raft_consensus.cc:374] T 00000000000000000000000000000000 P ae16374104dc4823a1362be92f41ed52 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 14:53:55.746995 12664 raft_consensus.cc:729] T 00000000000000000000000000000000 P ae16374104dc4823a1362be92f41ed52 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ae16374104dc4823a1362be92f41ed52, State: Initialized, Role: FOLLOWER
I20260927 14:53:55.747658 12664 consensus_queue.cc:260] T 00000000000000000000000000000000 P ae16374104dc4823a1362be92f41ed52 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae16374104dc4823a1362be92f41ed52" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42115 } }
I20260927 14:53:55.747851 12664 raft_consensus.cc:388] T 00000000000000000000000000000000 P ae16374104dc4823a1362be92f41ed52 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 14:53:55.747931 12664 raft_consensus.cc:482] T 00000000000000000000000000000000 P ae16374104dc4823a1362be92f41ed52 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 14:53:55.748049 12664 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ae16374104dc4823a1362be92f41ed52 [term 0 FOLLOWER]: Advancing to term 1
I20260927 14:53:55.749023 12667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:53:55.733183 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40954 (local address 127.0.98.62:42115)
0927 14:53:55.733633 (+ 450us) server_negotiation.cc:206] Beginning negotiation
0927 14:53:55.733651 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0927 14:53:55.733695 (+ 44us) server_negotiation.cc:407] Connection header received
0927 14:53:55.734529 (+ 834us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:53:55.734551 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:53:55.734963 (+ 412us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:53:55.735313 (+ 350us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:53:55.736217 (+ 904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:53:55.737754 (+ 1537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:53:55.738911 (+ 1157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:53:55.739601 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:53:55.739757 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:53:55.742770 (+ 3013us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:53:55.742777 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:53:55.742792 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 14:53:55.742822 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:53:55.745529 (+ 2707us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:53:55.746058 (+ 529us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:53:55.746064 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:53:55.746070 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:53:55.746235 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:53:55.746560 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:53:55.746563 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:53:55.746564 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:53:55.747107 (+ 543us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:53:55.747297 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0927 14:53:55.747697 (+ 400us) server_negotiation.cc:299] Negotiation successful
0927 14:53:55.747997 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":300,"thread_start_us":140,"threads_started":1}
I20260927 14:53:55.749234 12664 raft_consensus.cc:504] T 00000000000000000000000000000000 P ae16374104dc4823a1362be92f41ed52 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae16374104dc4823a1362be92f41ed52" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42115 } }
I20260927 14:53:55.749599 12664 leader_election.cc:302] T 00000000000000000000000000000000 P ae16374104dc4823a1362be92f41ed52 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ae16374104dc4823a1362be92f41ed52; no voters:
I20260927 14:53:55.749914 12664 leader_election.cc:288] T 00000000000000000000000000000000 P ae16374104dc4823a1362be92f41ed52 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 14:53:55.750074 12669 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ae16374104dc4823a1362be92f41ed52 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 14:53:55.750334 12669 raft_consensus.cc:686] T 00000000000000000000000000000000 P ae16374104dc4823a1362be92f41ed52 [term 1 LEADER]: Becoming Leader. State: Replica: ae16374104dc4823a1362be92f41ed52, State: Running, Role: LEADER
I20260927 14:53:55.750690 12669 consensus_queue.cc:237] T 00000000000000000000000000000000 P ae16374104dc4823a1362be92f41ed52 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae16374104dc4823a1362be92f41ed52" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42115 } }
I20260927 14:53:55.751170 12664 sys_catalog.cc:564] T 00000000000000000000000000000000 P ae16374104dc4823a1362be92f41ed52 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 14:53:55.752187 12670 sys_catalog.cc:455] T 00000000000000000000000000000000 P ae16374104dc4823a1362be92f41ed52 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ae16374104dc4823a1362be92f41ed52" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae16374104dc4823a1362be92f41ed52" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42115 } } }
I20260927 14:53:55.752300 12670 sys_catalog.cc:458] T 00000000000000000000000000000000 P ae16374104dc4823a1362be92f41ed52 [sys.catalog]: This master's current role is: LEADER
I20260927 14:53:55.752494 12670 sys_catalog.cc:455] T 00000000000000000000000000000000 P ae16374104dc4823a1362be92f41ed52 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ae16374104dc4823a1362be92f41ed52. Latest consensus state: current_term: 1 leader_uuid: "ae16374104dc4823a1362be92f41ed52" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae16374104dc4823a1362be92f41ed52" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42115 } } }
I20260927 14:53:55.752563 12670 sys_catalog.cc:458] T 00000000000000000000000000000000 P ae16374104dc4823a1362be92f41ed52 [sys.catalog]: This master's current role is: LEADER
I20260927 14:53:55.752681 12678 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 14:53:55.755888 12678 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 14:53:55.756321 12664 ranger_client.cc:318] Using new properties file: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790520684602251-392-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260927 14:53:55.774801 12678 catalog_manager.cc:1347] Generated new cluster ID: 1123e7bc5733449b9c9a968c4bfce270
I20260927 14:53:55.774886 12678 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 14:53:55.797550 12678 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 14:53:55.798974 12678 catalog_manager.cc:1504] Loading token signing keys...
I20260927 14:53:55.804584 12678 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ae16374104dc4823a1362be92f41ed52: Generated new TSK 0
I20260927 14:53:55.805559 12678 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790520684602251-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790520684602251-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260927 14:53:56.527419 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6936m5/build/debug/bin/kudu
/tmp/dist-test-task6936m5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790520684602251-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790520684602251-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790520684602251-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790520684602251-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:37591/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42115
--keytab_file=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790520684602251-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33451
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790520684602251-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790520684602251-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790520684602251-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 14:53:56.652549 12709 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 14:53:56.653028 12709 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 14:53:56.653184 12709 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 14:53:56.658577 12709 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 14:53:56.658665 12709 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 14:53:56.658762 12709 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 14:53:56.664466 12709 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33451
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790520684602251-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790520684602251-392-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.0.98.20:37591/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790520684602251-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790520684602251-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42115
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12709
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790520684602251-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:15:17 UTC on ccycloud.quasar-ledxfe.root.comops.site
build id 4676134
I20260927 14:53:56.665670 12709 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 14:53:56.666703 12709 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 14:53:56.676249 12717 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 14:53:56.676388 12716 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 14:53:56.676249 12719 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 14:53:56.676533 12709 server_base.cc:1034] running on GCE node
I20260927 14:53:56.677299 12709 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 14:53:56.677963 12709 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 14:53:56.679247 12709 hybrid_clock.cc:648] HybridClock initialized: now 1790520836679219 us; error 53 us; skew 500 ppm
Sep 27 14:53:56 dist-test-slave-xk1g krb5kdc[12152](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790520836, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 14:53:56.682175 12709 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 14:53:56.683212 12709 webserver.cc:460] Webserver started at http://127.0.98.1:39033/ using document root <none> and password file <none>
I20260927 14:53:56.683795 12709 fs_manager.cc:359] Metadata directory not provided
I20260927 14:53:56.683887 12709 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 14:53:56.684128 12709 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 27 14:53:56 dist-test-slave-xk1g krb5kdc[12152](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790520836, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260927 14:53:56.700925 12709 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790520684602251-392-0/minicluster-data/ts-0/data/instance:
uuid: "3c7df43e4dd84759ad0d79a4f1e4113f"
format_stamp: "Formatted at 2026-09-27 14:53:56 on dist-test-slave-xk1g"
server_key: "bb9c061e300a0195a2d7b4605594e9a0"
server_key_iv: "3212be3ac313452ae9caedcc3cd6248a"
server_key_version: "kuduclusterkey@0"
I20260927 14:53:56.702636 12709 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790520684602251-392-0/minicluster-data/ts-0/wal/instance:
uuid: "3c7df43e4dd84759ad0d79a4f1e4113f"
format_stamp: "Formatted at 2026-09-27 14:53:56 on dist-test-slave-xk1g"
server_key: "bb9c061e300a0195a2d7b4605594e9a0"
server_key_iv: "3212be3ac313452ae9caedcc3cd6248a"
server_key_version: "kuduclusterkey@0"
I20260927 14:53:56.707315 12709 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260927 14:53:56.729967 12726 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 14:53:56.731389 12709 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260927 14:53:56.731537 12709 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790520684602251-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790520684602251-392-0/minicluster-data/ts-0/wal
uuid: "3c7df43e4dd84759ad0d79a4f1e4113f"
format_stamp: "Formatted at 2026-09-27 14:53:56 on dist-test-slave-xk1g"
server_key: "bb9c061e300a0195a2d7b4605594e9a0"
server_key_iv: "3212be3ac313452ae9caedcc3cd6248a"
server_key_version: "kuduclusterkey@0"
I20260927 14:53:56.731680 12709 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790520684602251-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790520684602251-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790520684602251-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 14:53:56.746719 12709 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 14:53:56.751130 12709 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 14:53:56.751392 12709 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 14:53:56.754532 12709 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 14:53:56.755757 12709 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 14:53:56.755885 12709 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 14:53:56.756013 12709 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 14:53:56.756096 12709 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 14:53:56.767393 12709 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:41959
I20260927 14:53:56.767462 12839 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:41959 every 8 connection(s)
I20260927 14:53:56.768576 12709 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790520684602251-392-0/minicluster-data/ts-0/data/info.pb
I20260927 14:53:56.771767 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6936m5/build/debug/bin/kudu as pid 12709
I20260927 14:53:56.771880 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790520684602251-392-0/minicluster-data/ts-0/wal/instance
I20260927 14:53:56.772202 392 external_mini_cluster.cc:1428] Setting key 91b62c341a202bbf88fd9e4a7fbec38a
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Sep 27 14:53:56 dist-test-slave-xk1g krb5kdc[12152](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790520836, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 14:53:56.794627 12843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:53:56.771987 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42115 (local address 127.0.98.1:51519)
0927 14:53:56.772591 (+ 604us) negotiation.cc:107] Waiting for socket to connect
0927 14:53:56.772650 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0927 14:53:56.773742 (+ 1092us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:53:56.781299 (+ 7557us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:53:56.781310 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:53:56.781679 (+ 369us) client_negotiation.cc:189] Negotiated authn=SASL
0927 14:53:56.782408 (+ 729us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:53:56.782430 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:53:56.783437 (+ 1007us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:53:56.783440 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:53:56.784331 (+ 891us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:53:56.784341 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:53:56.784923 (+ 582us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:53:56.784927 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:53:56.785152 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:53:56.785848 (+ 696us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 14:53:56.785864 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 14:53:56.788552 (+ 2688us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 14:53:56.791268 (+ 2716us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 14:53:56.791274 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 14:53:56.791292 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0927 14:53:56.791566 (+ 274us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 14:53:56.791880 (+ 314us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 14:53:56.791883 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 14:53:56.791886 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 14:53:56.792037 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 14:53:56.792548 (+ 511us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 14:53:56.792555 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 14:53:56.792912 (+ 357us) client_negotiation.cc:769] Sending connection context
0927 14:53:56.793158 (+ 246us) client_negotiation.cc:240] Negotiation successful
0927 14:53:56.793528 (+ 370us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":355,"thread_start_us":140,"threads_started":1}
I20260927 14:53:56.795589 12842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:53:56.771439 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:51519 (local address 127.0.98.62:42115)
0927 14:53:56.780775 (+ 9336us) server_negotiation.cc:206] Beginning negotiation
0927 14:53:56.780784 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 14:53:56.780802 (+ 18us) server_negotiation.cc:407] Connection header received
0927 14:53:56.780849 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:53:56.780852 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:53:56.780901 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:53:56.782566 (+ 1665us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:53:56.782682 (+ 116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:53:56.783297 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:53:56.784466 (+ 1169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:53:56.784696 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:53:56.784866 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:53:56.788770 (+ 3904us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:53:56.788777 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:53:56.788783 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 14:53:56.788816 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:53:56.791140 (+ 2324us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:53:56.791682 (+ 542us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:53:56.791686 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:53:56.791687 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:53:56.791779 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:53:56.792169 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:53:56.792171 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:53:56.792172 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:53:56.792399 (+ 227us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:53:56.792492 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0927 14:53:56.795275 (+ 2783us) server_negotiation.cc:299] Negotiation successful
0927 14:53:56.795421 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9229,"thread_start_us":100,"threads_started":1}
I20260927 14:53:56.796880 12840 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42115
I20260927 14:53:56.797179 12840 heartbeater.cc:464] Registering TS with master...
I20260927 14:53:56.797788 12840 heartbeater.cc:511] Master 127.0.98.62:42115 requested a full tablet report, sending...
I20260927 14:53:56.799650 12628 ts_manager.cc:194] Registered new tserver with Master: 3c7df43e4dd84759ad0d79a4f1e4113f (127.0.98.1:41959)
I20260927 14:53:56.801286 12628 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:51519
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790520684602251-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790520684602251-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260927 14:53:56.822366 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6936m5/build/debug/bin/kudu
/tmp/dist-test-task6936m5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790520684602251-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790520684602251-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790520684602251-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790520684602251-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:37591/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42115
--keytab_file=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790520684602251-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33451
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790520684602251-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790520684602251-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790520684602251-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 14:53:56 dist-test-slave-xk1g krb5kdc[12152](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790520836, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260927 14:53:56.976696 12848 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 14:53:56.977007 12848 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 14:53:56.977110 12848 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 14:53:56.981541 12848 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 14:53:56.981694 12848 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 14:53:56.981819 12848 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 14:53:56.988391 12848 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33451
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790520684602251-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790520684602251-392-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.0.98.20:37591/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790520684602251-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790520684602251-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42115
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12848
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790520684602251-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:15:17 UTC on ccycloud.quasar-ledxfe.root.comops.site
build id 4676134
I20260927 14:53:56.989817 12848 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 14:53:56.991123 12848 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 14:53:57.003237 12855 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 14:53:57.003266 12857 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 14:53:57.003933 12848 server_base.cc:1034] running on GCE node
W20260927 14:53:57.004105 12854 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 14:53:57.004562 12848 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 14:53:57.005405 12848 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 14:53:57.006594 12848 hybrid_clock.cc:648] HybridClock initialized: now 1790520837006571 us; error 50 us; skew 500 ppm
Sep 27 14:53:57 dist-test-slave-xk1g krb5kdc[12152](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790520837, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 14:53:57.010150 12848 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 14:53:57.011296 12848 webserver.cc:460] Webserver started at http://127.0.98.2:36017/ using document root <none> and password file <none>
I20260927 14:53:57.012041 12848 fs_manager.cc:359] Metadata directory not provided
I20260927 14:53:57.012140 12848 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 14:53:57.012435 12848 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 27 14:53:57 dist-test-slave-xk1g krb5kdc[12152](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790520837, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260927 14:53:57.028529 12848 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790520684602251-392-0/minicluster-data/ts-1/data/instance:
uuid: "86a219d425c84c2eb7193910d77e413b"
format_stamp: "Formatted at 2026-09-27 14:53:57 on dist-test-slave-xk1g"
server_key: "37a81c1d74d916f5ae44de339ce4bb0a"
server_key_iv: "bff08ac1f6292bd2a54e3f318927d5fc"
server_key_version: "kuduclusterkey@0"
I20260927 14:53:57.029585 12848 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790520684602251-392-0/minicluster-data/ts-1/wal/instance:
uuid: "86a219d425c84c2eb7193910d77e413b"
format_stamp: "Formatted at 2026-09-27 14:53:57 on dist-test-slave-xk1g"
server_key: "37a81c1d74d916f5ae44de339ce4bb0a"
server_key_iv: "bff08ac1f6292bd2a54e3f318927d5fc"
server_key_version: "kuduclusterkey@0"
I20260927 14:53:57.037982 12848 fs_manager.cc:691] Time spent creating directory manager: real 0.008s user 0.004s sys 0.000s
I20260927 14:53:57.059701 12864 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 14:53:57.061450 12848 fs_manager.cc:725] Time spent opening block manager: real 0.009s user 0.002s sys 0.000s
I20260927 14:53:57.061606 12848 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790520684602251-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790520684602251-392-0/minicluster-data/ts-1/wal
uuid: "86a219d425c84c2eb7193910d77e413b"
format_stamp: "Formatted at 2026-09-27 14:53:57 on dist-test-slave-xk1g"
server_key: "37a81c1d74d916f5ae44de339ce4bb0a"
server_key_iv: "bff08ac1f6292bd2a54e3f318927d5fc"
server_key_version: "kuduclusterkey@0"
I20260927 14:53:57.061756 12848 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790520684602251-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790520684602251-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790520684602251-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 14:53:57.079017 12848 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 14:53:57.082494 12848 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 14:53:57.082685 12848 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 14:53:57.083334 12848 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 14:53:57.084401 12848 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 14:53:57.084462 12848 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 14:53:57.084518 12848 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 14:53:57.084548 12848 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 14:53:57.097638 12848 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:36931
I20260927 14:53:57.098834 12848 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790520684602251-392-0/minicluster-data/ts-1/data/info.pb
I20260927 14:53:57.099706 12977 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:36931 every 8 connection(s)
I20260927 14:53:57.104539 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6936m5/build/debug/bin/kudu as pid 12848
I20260927 14:53:57.104674 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790520684602251-392-0/minicluster-data/ts-1/wal/instance
I20260927 14:53:57.104977 392 external_mini_cluster.cc:1428] Setting key 1d8236375ef33cdf846ef419b6ce9120
Sep 27 14:53:57 dist-test-slave-xk1g krb5kdc[12152](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790520837, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 14:53:57.121467 12842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:53:57.103222 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:47559 (local address 127.0.98.62:42115)
0927 14:53:57.103348 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0927 14:53:57.103353 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 14:53:57.104499 (+ 1146us) server_negotiation.cc:407] Connection header received
0927 14:53:57.106875 (+ 2376us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:53:57.106881 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:53:57.106941 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:53:57.106998 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:53:57.108725 (+ 1727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:53:57.109567 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:53:57.111259 (+ 1692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:53:57.111610 (+ 351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:53:57.111684 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:53:57.117156 (+ 5472us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:53:57.117160 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:53:57.117163 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 14:53:57.117191 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:53:57.118999 (+ 1808us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:53:57.119613 (+ 614us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:53:57.119617 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:53:57.119619 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:53:57.119700 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:53:57.120101 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:53:57.120104 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:53:57.120106 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:53:57.120310 (+ 204us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:53:57.120380 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0927 14:53:57.121144 (+ 764us) server_negotiation.cc:299] Negotiation successful
0927 14:53:57.121279 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260927 14:53:57.123260 12980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:53:57.102141 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42115 (local address 127.0.98.2:47559)
0927 14:53:57.104267 (+ 2126us) negotiation.cc:107] Waiting for socket to connect
0927 14:53:57.104387 (+ 120us) client_negotiation.cc:174] Beginning negotiation
0927 14:53:57.106635 (+ 2248us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:53:57.107428 (+ 793us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:53:57.107441 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:53:57.107804 (+ 363us) client_negotiation.cc:189] Negotiated authn=SASL
0927 14:53:57.108503 (+ 699us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:53:57.108521 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:53:57.109721 (+ 1200us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:53:57.109724 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:53:57.111105 (+ 1381us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:53:57.111115 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:53:57.111871 (+ 756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:53:57.111874 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:53:57.112062 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:53:57.115241 (+ 3179us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 14:53:57.115263 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0927 14:53:57.116924 (+ 1661us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 14:53:57.119154 (+ 2230us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 14:53:57.119160 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 14:53:57.119176 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0927 14:53:57.119498 (+ 322us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 14:53:57.119805 (+ 307us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 14:53:57.119808 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 14:53:57.119810 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 14:53:57.119970 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 14:53:57.120486 (+ 516us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 14:53:57.120492 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 14:53:57.120813 (+ 321us) client_negotiation.cc:769] Sending connection context
0927 14:53:57.121906 (+ 1093us) client_negotiation.cc:240] Negotiation successful
0927 14:53:57.122193 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1916,"thread_start_us":126,"threads_started":1}
I20260927 14:53:57.124828 12978 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42115
I20260927 14:53:57.125089 12978 heartbeater.cc:464] Registering TS with master...
I20260927 14:53:57.125674 12978 heartbeater.cc:511] Master 127.0.98.62:42115 requested a full tablet report, sending...
I20260927 14:53:57.126821 12628 ts_manager.cc:194] Registered new tserver with Master: 86a219d425c84c2eb7193910d77e413b (127.0.98.2:36931)
I20260927 14:53:57.127539 12628 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:47559
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790520684602251-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790520684602251-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260927 14:53:57.158489 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6936m5/build/debug/bin/kudu
/tmp/dist-test-task6936m5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790520684602251-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790520684602251-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790520684602251-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790520684602251-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:37591/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42115
--keytab_file=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790520684602251-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33451
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790520684602251-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790520684602251-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790520684602251-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 14:53:57.336645 12983 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 14:53:57.337162 12983 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 14:53:57.337260 12983 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 14:53:57.343462 12983 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 14:53:57.343555 12983 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 14:53:57.343657 12983 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 14:53:57.351931 12983 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33451
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790520684602251-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790520684602251-392-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.0.98.20:37591/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790520684602251-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790520684602251-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42115
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12983
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790520684602251-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:15:17 UTC on ccycloud.quasar-ledxfe.root.comops.site
build id 4676134
I20260927 14:53:57.353562 12983 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 14:53:57.355077 12983 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 14:53:57.368173 12989 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 14:53:57.368206 12988 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 14:53:57.369338 12983 server_base.cc:1034] running on GCE node
W20260927 14:53:57.369639 12991 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 14:53:57.370141 12983 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 14:53:57.370960 12983 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 14:53:57.372129 12983 hybrid_clock.cc:648] HybridClock initialized: now 1790520837372058 us; error 88 us; skew 500 ppm
Sep 27 14:53:57 dist-test-slave-xk1g krb5kdc[12152](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790520837, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 14:53:57.375988 12983 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 14:53:57.377206 12983 webserver.cc:460] Webserver started at http://127.0.98.3:38745/ using document root <none> and password file <none>
I20260927 14:53:57.377990 12983 fs_manager.cc:359] Metadata directory not provided
I20260927 14:53:57.378155 12983 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 14:53:57.378461 12983 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 27 14:53:57 dist-test-slave-xk1g krb5kdc[12152](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790520837, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260927 14:53:57.399058 12983 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790520684602251-392-0/minicluster-data/ts-2/data/instance:
uuid: "80493bf624dc4fd7b434bd8beadf232d"
format_stamp: "Formatted at 2026-09-27 14:53:57 on dist-test-slave-xk1g"
server_key: "29b955ce5d5332f77b5a22b8dd9541ac"
server_key_iv: "d0cdd4227b03893f535ea22adce649e2"
server_key_version: "kuduclusterkey@0"
I20260927 14:53:57.403311 12983 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790520684602251-392-0/minicluster-data/ts-2/wal/instance:
uuid: "80493bf624dc4fd7b434bd8beadf232d"
format_stamp: "Formatted at 2026-09-27 14:53:57 on dist-test-slave-xk1g"
server_key: "29b955ce5d5332f77b5a22b8dd9541ac"
server_key_iv: "d0cdd4227b03893f535ea22adce649e2"
server_key_version: "kuduclusterkey@0"
Sep 27 14:53:57 dist-test-slave-xk1g krb5kdc[12152](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790520836, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260927 14:53:57.422570 12983 fs_manager.cc:691] Time spent creating directory manager: real 0.019s user 0.000s sys 0.004s
I20260927 14:53:57.457391 12998 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 14:53:57.459127 12983 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260927 14:53:57.459261 12983 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790520684602251-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790520684602251-392-0/minicluster-data/ts-2/wal
uuid: "80493bf624dc4fd7b434bd8beadf232d"
format_stamp: "Formatted at 2026-09-27 14:53:57 on dist-test-slave-xk1g"
server_key: "29b955ce5d5332f77b5a22b8dd9541ac"
server_key_iv: "d0cdd4227b03893f535ea22adce649e2"
server_key_version: "kuduclusterkey@0"
I20260927 14:53:57.459389 12983 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790520684602251-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790520684602251-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790520684602251-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 14:53:57.474987 12983 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 14:53:57.488180 12983 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 14:53:57.488399 12983 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 14:53:57.489396 12983 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 14:53:57.490487 12983 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 14:53:57.490549 12983 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 14:53:57.490608 12983 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 14:53:57.490643 12983 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 14:53:57.504333 12983 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:45701
I20260927 14:53:57.505512 12983 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790520684602251-392-0/minicluster-data/ts-2/data/info.pb
I20260927 14:53:57.506217 13111 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:45701 every 8 connection(s)
I20260927 14:53:57.512506 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6936m5/build/debug/bin/kudu as pid 12983
I20260927 14:53:57.512632 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790520684602251-392-0/minicluster-data/ts-2/wal/instance
I20260927 14:53:57.512912 392 external_mini_cluster.cc:1428] Setting key 03937fe4777918dd51700892f7bf6b86
Sep 27 14:53:57 dist-test-slave-xk1g krb5kdc[12152](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790520837, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 14:53:57.524801 12842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:53:57.508343 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:57523 (local address 127.0.98.62:42115)
0927 14:53:57.508479 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0927 14:53:57.508484 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 14:53:57.510401 (+ 1917us) server_negotiation.cc:407] Connection header received
0927 14:53:57.511523 (+ 1122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:53:57.511528 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:53:57.511597 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:53:57.511698 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:53:57.514520 (+ 2822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:53:57.515294 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:53:57.517004 (+ 1710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:53:57.517361 (+ 357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:53:57.517435 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:53:57.520472 (+ 3037us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:53:57.520477 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:53:57.520479 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 14:53:57.520509 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:53:57.522326 (+ 1817us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:53:57.522966 (+ 640us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:53:57.522970 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:53:57.522972 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:53:57.523056 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:53:57.523475 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:53:57.523479 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:53:57.523481 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:53:57.523697 (+ 216us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:53:57.523790 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0927 14:53:57.524506 (+ 716us) server_negotiation.cc:299] Negotiation successful
0927 14:53:57.524636 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260927 14:53:57.527676 13114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:53:57.508890 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42115 (local address 127.0.98.3:57523)
0927 14:53:57.510240 (+ 1350us) negotiation.cc:107] Waiting for socket to connect
0927 14:53:57.510294 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0927 14:53:57.511297 (+ 1003us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:53:57.511990 (+ 693us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:53:57.511999 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:53:57.513472 (+ 1473us) client_negotiation.cc:189] Negotiated authn=SASL
0927 14:53:57.514348 (+ 876us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:53:57.514374 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:53:57.515512 (+ 1138us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:53:57.515515 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:53:57.516866 (+ 1351us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:53:57.516877 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:53:57.517633 (+ 756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:53:57.517636 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:53:57.517827 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:53:57.518506 (+ 679us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 14:53:57.518527 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0927 14:53:57.520259 (+ 1732us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 14:53:57.522471 (+ 2212us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 14:53:57.522477 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 14:53:57.522496 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0927 14:53:57.522849 (+ 353us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 14:53:57.523178 (+ 329us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 14:53:57.523181 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 14:53:57.523183 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 14:53:57.523363 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 14:53:57.523880 (+ 517us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 14:53:57.523888 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 14:53:57.524222 (+ 334us) client_negotiation.cc:769] Sending connection context
0927 14:53:57.524990 (+ 768us) client_negotiation.cc:240] Negotiation successful
0927 14:53:57.525379 (+ 389us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1122,"thread_start_us":204,"threads_started":1}
I20260927 14:53:57.528781 13112 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42115
I20260927 14:53:57.529042 13112 heartbeater.cc:464] Registering TS with master...
I20260927 14:53:57.529670 13112 heartbeater.cc:511] Master 127.0.98.62:42115 requested a full tablet report, sending...
I20260927 14:53:57.530751 12628 ts_manager.cc:194] Registered new tserver with Master: 80493bf624dc4fd7b434bd8beadf232d (127.0.98.3:45701)
I20260927 14:53:57.531490 12628 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:57523
I20260927 14:53:57.538734 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 14:53:57.539515 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6936m5/build/debug/bin/kudu with pid 12709
I20260927 14:53:57.547672 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6936m5/build/debug/bin/kudu with pid 12848
I20260927 14:53:57.555421 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6936m5/build/debug/bin/kudu with pid 12983
I20260927 14:53:57.564318 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6936m5/build/debug/bin/kudu with pid 12596
I20260927 14:53:57.572770 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6936m5/build/debug/bin/kudu
/tmp/dist-test-task6936m5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790520684602251-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790520684602251-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790520684602251-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790520684602251-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42115
--webserver_interface=127.0.98.62
--webserver_port=46313
--keytab_file=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790520684602251-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33451
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:42115
--ranger_config_path=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790520684602251-392-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:37591/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:37591/kms with env {KRB5CCNAME=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790520684602251-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790520684602251-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790520684602251-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 14:53:57.684840 13115 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 14:53:57.685222 13115 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 14:53:57.685307 13115 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 14:53:57.689316 13115 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 14:53:57.689373 13115 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 14:53:57.689392 13115 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 14:53:57.689409 13115 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 14:53:57.689424 13115 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 14:53:57.695251 13115 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33451
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790520684602251-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790520684602251-392-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:37591/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:42115
--ranger_config_path=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790520684602251-392-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790520684602251-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42115
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790520684602251-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=46313
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13115
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790520684602251-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:15:17 UTC on ccycloud.quasar-ledxfe.root.comops.site
build id 4676134
I20260927 14:53:57.696341 13115 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 14:53:57.697407 13115 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 14:53:57.705814 13120 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 14:53:57.705850 13115 server_base.cc:1034] running on GCE node
W20260927 14:53:57.705826 13123 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 14:53:57.705814 13121 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 14:53:57.706627 13115 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 14:53:57.707687 13115 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 14:53:57.708846 13115 hybrid_clock.cc:648] HybridClock initialized: now 1790520837708813 us; error 49 us; skew 500 ppm
Sep 27 14:53:57 dist-test-slave-xk1g krb5kdc[12152](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790520837, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 14:53:57.711685 13115 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 14:53:57.712570 13115 webserver.cc:460] Webserver started at http://127.0.98.62:46313/ using document root <none> and password file <none>
I20260927 14:53:57.713209 13115 fs_manager.cc:359] Metadata directory not provided
I20260927 14:53:57.713281 13115 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 27 14:53:57 dist-test-slave-xk1g krb5kdc[12152](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790520837, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260927 14:53:57.756673 13115 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260927 14:53:57.759393 13131 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 14:53:57.760928 13115 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260927 14:53:57.761109 13115 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790520684602251-392-0/minicluster-data/master-0/data,/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790520684602251-392-0/minicluster-data/master-0/wal
uuid: "ae16374104dc4823a1362be92f41ed52"
format_stamp: "Formatted at 2026-09-27 14:53:55 on dist-test-slave-xk1g"
server_key: "b417a96899809b6696e1d87e4609327d"
server_key_iv: "690a343ff21ebd64374fba4001984df6"
server_key_version: "kuduclusterkey@0"
I20260927 14:53:57.761649 13115 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790520684602251-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790520684602251-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790520684602251-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 14:53:57.775731 13115 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 14:53:57.778671 13115 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 14:53:57.778870 13115 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 14:53:57.785794 13115 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:42115
I20260927 14:53:57.785800 13183 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:42115 every 8 connection(s)
I20260927 14:53:57.786834 13115 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790520684602251-392-0/minicluster-data/master-0/data/info.pb
I20260927 14:53:57.790969 13184 sys_catalog.cc:263] Verifying existing consensus state
I20260927 14:53:57.791735 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6936m5/build/debug/bin/kudu as pid 13115
I20260927 14:53:57.792232 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6936m5/build/debug/bin/kudu
/tmp/dist-test-task6936m5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790520684602251-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790520684602251-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790520684602251-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790520684602251-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:37591/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:41959
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:42115
--webserver_port=39033
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790520684602251-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33451
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:37591/kms with env {KRB5CCNAME=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790520684602251-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790520684602251-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790520684602251-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 14:53:57.792963 13184 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ae16374104dc4823a1362be92f41ed52: Bootstrap starting.
I20260927 14:53:57.809710 13184 log.cc:824] T 00000000000000000000000000000000 P ae16374104dc4823a1362be92f41ed52: Log is configured to *not* fsync() on all Append() calls
I20260927 14:53:57.816278 13184 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ae16374104dc4823a1362be92f41ed52: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 14:53:57.816675 13184 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ae16374104dc4823a1362be92f41ed52: Bootstrap complete.
I20260927 14:53:57.820680 13184 raft_consensus.cc:348] T 00000000000000000000000000000000 P ae16374104dc4823a1362be92f41ed52 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae16374104dc4823a1362be92f41ed52" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42115 } }
I20260927 14:53:57.821108 13184 raft_consensus.cc:729] T 00000000000000000000000000000000 P ae16374104dc4823a1362be92f41ed52 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: ae16374104dc4823a1362be92f41ed52, State: Initialized, Role: FOLLOWER
I20260927 14:53:57.821568 13184 consensus_queue.cc:260] T 00000000000000000000000000000000 P ae16374104dc4823a1362be92f41ed52 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae16374104dc4823a1362be92f41ed52" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42115 } }
I20260927 14:53:57.821722 13184 raft_consensus.cc:388] T 00000000000000000000000000000000 P ae16374104dc4823a1362be92f41ed52 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 14:53:57.821800 13184 raft_consensus.cc:482] T 00000000000000000000000000000000 P ae16374104dc4823a1362be92f41ed52 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 14:53:57.821916 13184 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ae16374104dc4823a1362be92f41ed52 [term 1 FOLLOWER]: Advancing to term 2
I20260927 14:53:57.823657 13184 raft_consensus.cc:504] T 00000000000000000000000000000000 P ae16374104dc4823a1362be92f41ed52 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae16374104dc4823a1362be92f41ed52" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42115 } }
I20260927 14:53:57.823998 13184 leader_election.cc:302] T 00000000000000000000000000000000 P ae16374104dc4823a1362be92f41ed52 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ae16374104dc4823a1362be92f41ed52; no voters:
I20260927 14:53:57.824326 13184 leader_election.cc:288] T 00000000000000000000000000000000 P ae16374104dc4823a1362be92f41ed52 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260927 14:53:57.824442 13188 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ae16374104dc4823a1362be92f41ed52 [term 2 FOLLOWER]: Leader election won for term 2
I20260927 14:53:57.824703 13188 raft_consensus.cc:686] T 00000000000000000000000000000000 P ae16374104dc4823a1362be92f41ed52 [term 2 LEADER]: Becoming Leader. State: Replica: ae16374104dc4823a1362be92f41ed52, State: Running, Role: LEADER
I20260927 14:53:57.824992 13188 consensus_queue.cc:237] T 00000000000000000000000000000000 P ae16374104dc4823a1362be92f41ed52 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae16374104dc4823a1362be92f41ed52" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42115 } }
I20260927 14:53:57.825485 13184 sys_catalog.cc:564] T 00000000000000000000000000000000 P ae16374104dc4823a1362be92f41ed52 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 14:53:57.826287 13190 sys_catalog.cc:455] T 00000000000000000000000000000000 P ae16374104dc4823a1362be92f41ed52 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ae16374104dc4823a1362be92f41ed52. Latest consensus state: current_term: 2 leader_uuid: "ae16374104dc4823a1362be92f41ed52" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae16374104dc4823a1362be92f41ed52" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42115 } } }
I20260927 14:53:57.826411 13190 sys_catalog.cc:458] T 00000000000000000000000000000000 P ae16374104dc4823a1362be92f41ed52 [sys.catalog]: This master's current role is: LEADER
I20260927 14:53:57.826726 13189 sys_catalog.cc:455] T 00000000000000000000000000000000 P ae16374104dc4823a1362be92f41ed52 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "ae16374104dc4823a1362be92f41ed52" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae16374104dc4823a1362be92f41ed52" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42115 } } }
I20260927 14:53:57.826812 13189 sys_catalog.cc:458] T 00000000000000000000000000000000 P ae16374104dc4823a1362be92f41ed52 [sys.catalog]: This master's current role is: LEADER
I20260927 14:53:57.827386 13197 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 14:53:57.827670 13184 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790520684602251-392-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260927 14:53:57.832867 13197 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 14:53:57.833554 13197 catalog_manager.cc:1259] Loaded cluster ID: 1123e7bc5733449b9c9a968c4bfce270
I20260927 14:53:57.833617 13197 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 14:53:57.836099 13197 catalog_manager.cc:1504] Loading token signing keys...
I20260927 14:53:57.836802 13197 catalog_manager.cc:5879] T 00000000000000000000000000000000 P ae16374104dc4823a1362be92f41ed52: Loaded TSK: 0
I20260927 14:53:57.838033 13197 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260927 14:53:57.912019 13186 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 14:53:57.912300 13186 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 14:53:57.912376 13186 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 14:53:57.917879 13186 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 14:53:57.917958 13186 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 14:53:57.918063 13186 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 14:53:57.923341 13186 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33451
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790520684602251-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790520684602251-392-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:37591/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790520684602251-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:41959
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790520684602251-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=39033
--tserver_master_addrs=127.0.98.62:42115
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13186
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790520684602251-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:15:17 UTC on ccycloud.quasar-ledxfe.root.comops.site
build id 4676134
I20260927 14:53:57.924556 13186 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 14:53:57.925822 13186 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 14:53:57.937745 13224 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 14:53:57.937875 13221 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 14:53:57.938172 13186 server_base.cc:1034] running on GCE node
W20260927 14:53:57.937758 13222 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 14:53:57.938887 13186 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 14:53:57.939574 13186 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 14:53:57.940765 13186 hybrid_clock.cc:648] HybridClock initialized: now 1790520837940730 us; error 46 us; skew 500 ppm
Sep 27 14:53:57 dist-test-slave-xk1g krb5kdc[12152](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790520837, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 14:53:57.943645 13186 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 14:53:57.944654 13186 webserver.cc:460] Webserver started at http://127.0.98.1:39033/ using document root <none> and password file <none>
I20260927 14:53:57.945350 13186 fs_manager.cc:359] Metadata directory not provided
I20260927 14:53:57.945461 13186 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 27 14:53:57 dist-test-slave-xk1g krb5kdc[12152](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790520837, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260927 14:53:57.986220 13186 fs_manager.cc:709] Time spent opening directory manager: real 0.015s user 0.002s sys 0.000s
I20260927 14:53:57.989678 13232 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 14:53:57.991194 13186 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260927 14:53:57.991321 13186 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790520684602251-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790520684602251-392-0/minicluster-data/ts-0/wal
uuid: "3c7df43e4dd84759ad0d79a4f1e4113f"
format_stamp: "Formatted at 2026-09-27 14:53:56 on dist-test-slave-xk1g"
server_key: "bb9c061e300a0195a2d7b4605594e9a0"
server_key_iv: "3212be3ac313452ae9caedcc3cd6248a"
server_key_version: "kuduclusterkey@0"
I20260927 14:53:57.992069 13186 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790520684602251-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790520684602251-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790520684602251-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 14:53:58.033350 13186 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 14:53:58.037173 13186 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 14:53:58.037387 13186 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 14:53:58.038117 13186 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 14:53:58.039158 13186 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 14:53:58.039222 13186 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 14:53:58.039285 13186 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 14:53:58.039309 13186 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 14:53:58.049671 13186 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:41959
I20260927 14:53:58.051015 13186 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790520684602251-392-0/minicluster-data/ts-0/data/info.pb
I20260927 14:53:58.055475 13345 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:41959 every 8 connection(s)
I20260927 14:53:58.056650 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6936m5/build/debug/bin/kudu as pid 13186
I20260927 14:53:58.057264 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6936m5/build/debug/bin/kudu
/tmp/dist-test-task6936m5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790520684602251-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790520684602251-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790520684602251-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790520684602251-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:37591/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:36931
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:42115
--webserver_port=36017
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790520684602251-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33451
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:37591/kms with env {KRB5CCNAME=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790520684602251-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790520684602251-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790520684602251-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 14:53:58 dist-test-slave-xk1g krb5kdc[12152](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790520837, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 14:53:58.079627 13348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:53:58.057944 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42115 (local address 127.0.98.1:42355)
0927 14:53:58.061149 (+ 3205us) negotiation.cc:107] Waiting for socket to connect
0927 14:53:58.061220 (+ 71us) client_negotiation.cc:174] Beginning negotiation
0927 14:53:58.062309 (+ 1089us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:53:58.063917 (+ 1608us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:53:58.063926 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:53:58.064288 (+ 362us) client_negotiation.cc:189] Negotiated authn=SASL
0927 14:53:58.064952 (+ 664us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:53:58.064973 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:53:58.067166 (+ 2193us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:53:58.067172 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:53:58.068814 (+ 1642us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:53:58.068828 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:53:58.070398 (+ 1570us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:53:58.070404 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:53:58.070639 (+ 235us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:53:58.071425 (+ 786us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 14:53:58.071452 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0927 14:53:58.073248 (+ 1796us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 14:53:58.075839 (+ 2591us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 14:53:58.075846 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 14:53:58.075863 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0927 14:53:58.076348 (+ 485us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 14:53:58.076744 (+ 396us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 14:53:58.076749 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 14:53:58.076751 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 14:53:58.076934 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 14:53:58.077727 (+ 793us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 14:53:58.077734 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 14:53:58.078048 (+ 314us) client_negotiation.cc:769] Sending connection context
0927 14:53:58.078238 (+ 190us) client_negotiation.cc:240] Negotiation successful
0927 14:53:58.078578 (+ 340us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2940,"thread_start_us":154,"threads_started":1}
I20260927 14:53:58.081595 13350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:53:58.059471 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:42355 (local address 127.0.98.62:42115)
0927 14:53:58.060080 (+ 609us) server_negotiation.cc:206] Beginning negotiation
0927 14:53:58.060105 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0927 14:53:58.062651 (+ 2546us) server_negotiation.cc:407] Connection header received
0927 14:53:58.062975 (+ 324us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:53:58.062991 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:53:58.063410 (+ 419us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:53:58.063803 (+ 393us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:53:58.065161 (+ 1358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:53:58.066960 (+ 1799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:53:58.068987 (+ 2027us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:53:58.069918 (+ 931us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:53:58.070121 (+ 203us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:53:58.073512 (+ 3391us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:53:58.073521 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:53:58.073540 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 14:53:58.073574 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:53:58.075702 (+ 2128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:53:58.076488 (+ 786us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:53:58.076495 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:53:58.076501 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:53:58.076646 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:53:58.077044 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:53:58.077047 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:53:58.077049 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:53:58.077549 (+ 500us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:53:58.077674 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0927 14:53:58.080306 (+ 2632us) server_negotiation.cc:299] Negotiation successful
0927 14:53:58.080648 (+ 342us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":426,"thread_start_us":123,"threads_started":1}
I20260927 14:53:58.084879 13346 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42115
I20260927 14:53:58.085155 13346 heartbeater.cc:464] Registering TS with master...
I20260927 14:53:58.085717 13346 heartbeater.cc:511] Master 127.0.98.62:42115 requested a full tablet report, sending...
W20260927 14:53:58.087880 13346 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:42115 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260927 14:53:58.087999 13346 heartbeater.cc:464] Registering TS with master...
I20260927 14:53:58.088172 13346 heartbeater.cc:511] Master 127.0.98.62:42115 requested a full tablet report, sending...
I20260927 14:53:58.088979 13346 heartbeater.cc:464] Registering TS with master...
I20260927 14:53:58.089178 13346 heartbeater.cc:511] Master 127.0.98.62:42115 requested a full tablet report, sending...
W20260927 14:53:58.089884 13346 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260927 14:53:58.206094 13349 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 14:53:58.206481 13349 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 14:53:58.206563 13349 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 14:53:58.211719 13349 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 14:53:58.211793 13349 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 14:53:58.211855 13349 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 14:53:58.218269 13349 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33451
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790520684602251-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790520684602251-392-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:37591/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790520684602251-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:36931
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790520684602251-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=36017
--tserver_master_addrs=127.0.98.62:42115
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13349
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790520684602251-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:15:17 UTC on ccycloud.quasar-ledxfe.root.comops.site
build id 4676134
I20260927 14:53:58.219645 13349 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 14:53:58.220927 13349 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 14:53:58.231437 13356 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 14:53:58.231453 13355 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 14:53:58.231623 13349 server_base.cc:1034] running on GCE node
W20260927 14:53:58.231456 13358 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 14:53:58.232155 13349 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 14:53:58.232755 13349 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 14:53:58.233942 13349 hybrid_clock.cc:648] HybridClock initialized: now 1790520838233899 us; error 58 us; skew 500 ppm
Sep 27 14:53:58 dist-test-slave-xk1g krb5kdc[12152](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790520838, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 14:53:58.237179 13349 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 14:53:58.238112 13349 webserver.cc:460] Webserver started at http://127.0.98.2:36017/ using document root <none> and password file <none>
I20260927 14:53:58.238682 13349 fs_manager.cc:359] Metadata directory not provided
I20260927 14:53:58.238786 13349 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 27 14:53:58 dist-test-slave-xk1g krb5kdc[12152](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790520838, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260927 14:53:58.270140 13349 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.000s
I20260927 14:53:58.273290 13366 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 14:53:58.274847 13349 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 14:53:58.274984 13349 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790520684602251-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790520684602251-392-0/minicluster-data/ts-1/wal
uuid: "86a219d425c84c2eb7193910d77e413b"
format_stamp: "Formatted at 2026-09-27 14:53:57 on dist-test-slave-xk1g"
server_key: "37a81c1d74d916f5ae44de339ce4bb0a"
server_key_iv: "bff08ac1f6292bd2a54e3f318927d5fc"
server_key_version: "kuduclusterkey@0"
I20260927 14:53:58.275704 13349 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790520684602251-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790520684602251-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790520684602251-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 14:53:58.295738 13349 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 14:53:58.299396 13349 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 14:53:58.299597 13349 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 14:53:58.300328 13349 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 14:53:58.301400 13349 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 14:53:58.301468 13349 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 14:53:58.301522 13349 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 14:53:58.301555 13349 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 14:53:58.317422 13349 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:36931
I20260927 14:53:58.317445 13479 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:36931 every 8 connection(s)
I20260927 14:53:58.318783 13349 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790520684602251-392-0/minicluster-data/ts-1/data/info.pb
I20260927 14:53:58.320664 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6936m5/build/debug/bin/kudu as pid 13349
I20260927 14:53:58.321130 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6936m5/build/debug/bin/kudu
/tmp/dist-test-task6936m5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790520684602251-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790520684602251-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790520684602251-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790520684602251-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:37591/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:45701
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:42115
--webserver_port=38745
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790520684602251-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33451
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:37591/kms with env {KRB5CCNAME=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790520684602251-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790520684602251-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790520684602251-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 14:53:58 dist-test-slave-xk1g krb5kdc[12152](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790520838, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 14:53:58.352919 13350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:53:58.322526 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:47445 (local address 127.0.98.62:42115)
0927 14:53:58.322646 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0927 14:53:58.322651 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 14:53:58.322667 (+ 16us) server_negotiation.cc:407] Connection header received
0927 14:53:58.323509 (+ 842us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:53:58.323515 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:53:58.323582 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:53:58.323640 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:53:58.325146 (+ 1506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:53:58.326033 (+ 887us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:53:58.328175 (+ 2142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:53:58.328527 (+ 352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:53:58.328620 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:53:58.340401 (+ 11781us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:53:58.340406 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:53:58.340408 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 14:53:58.340439 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:53:58.349945 (+ 9506us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:53:58.350763 (+ 818us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:53:58.350769 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:53:58.350773 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:53:58.350920 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:53:58.351394 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:53:58.351397 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:53:58.351399 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:53:58.351659 (+ 260us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:53:58.351750 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0927 14:53:58.352597 (+ 847us) server_negotiation.cc:299] Negotiation successful
0927 14:53:58.352746 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260927 14:53:58.358831 13482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:53:58.321428 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42115 (local address 127.0.98.2:47445)
0927 14:53:58.321986 (+ 558us) negotiation.cc:107] Waiting for socket to connect
0927 14:53:58.322068 (+ 82us) client_negotiation.cc:174] Beginning negotiation
0927 14:53:58.323230 (+ 1162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:53:58.324109 (+ 879us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:53:58.324120 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:53:58.324489 (+ 369us) client_negotiation.cc:189] Negotiated authn=SASL
0927 14:53:58.324963 (+ 474us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:53:58.324978 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:53:58.326179 (+ 1201us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:53:58.326183 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:53:58.328023 (+ 1840us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:53:58.328035 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:53:58.328840 (+ 805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:53:58.328844 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:53:58.328987 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:53:58.337793 (+ 8806us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 14:53:58.337817 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0927 14:53:58.340157 (+ 2340us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 14:53:58.350138 (+ 9981us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 14:53:58.350147 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 14:53:58.350169 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0927 14:53:58.350621 (+ 452us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 14:53:58.351050 (+ 429us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 14:53:58.351053 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 14:53:58.351055 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 14:53:58.351274 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 14:53:58.351824 (+ 550us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 14:53:58.351832 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 14:53:58.352282 (+ 450us) client_negotiation.cc:769] Sending connection context
0927 14:53:58.353204 (+ 922us) client_negotiation.cc:240] Negotiation successful
0927 14:53:58.353600 (+ 396us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":305,"thread_start_us":145,"threads_started":1}
I20260927 14:53:58.371959 13480 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42115
I20260927 14:53:58.372243 13480 heartbeater.cc:464] Registering TS with master...
I20260927 14:53:58.372874 13480 heartbeater.cc:511] Master 127.0.98.62:42115 requested a full tablet report, sending...
W20260927 14:53:58.374722 13480 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:42115 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260927 14:53:58.374862 13480 heartbeater.cc:464] Registering TS with master...
I20260927 14:53:58.375087 13480 heartbeater.cc:511] Master 127.0.98.62:42115 requested a full tablet report, sending...
I20260927 14:53:58.376050 13480 heartbeater.cc:464] Registering TS with master...
I20260927 14:53:58.376257 13480 heartbeater.cc:511] Master 127.0.98.62:42115 requested a full tablet report, sending...
W20260927 14:53:58.377148 13480 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260927 14:53:58.492570 13483 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 14:53:58.492950 13483 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 14:53:58.493024 13483 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 14:53:58.499143 13483 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 14:53:58.499228 13483 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 14:53:58.499328 13483 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 14:53:58.507457 13483 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33451
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790520684602251-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790520684602251-392-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:37591/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790520684602251-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:45701
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790520684602251-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=38745
--tserver_master_addrs=127.0.98.62:42115
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13483
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790520684602251-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:15:17 UTC on ccycloud.quasar-ledxfe.root.comops.site
build id 4676134
I20260927 14:53:58.508908 13483 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 14:53:58.510273 13483 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 14:53:58.522351 13492 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 14:53:58.522359 13489 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 14:53:58.522917 13490 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 14:53:58.523634 13483 server_base.cc:1034] running on GCE node
I20260927 14:53:58.524217 13483 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 14:53:58.525005 13483 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 14:53:58.526234 13483 hybrid_clock.cc:648] HybridClock initialized: now 1790520838526209 us; error 41 us; skew 500 ppm
Sep 27 14:53:58 dist-test-slave-xk1g krb5kdc[12152](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790520838, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 14:53:58.529790 13483 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 14:53:58.530683 13483 webserver.cc:460] Webserver started at http://127.0.98.3:38745/ using document root <none> and password file <none>
I20260927 14:53:58.531261 13483 fs_manager.cc:359] Metadata directory not provided
I20260927 14:53:58.531344 13483 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 27 14:53:58 dist-test-slave-xk1g krb5kdc[12152](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790520838, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260927 14:53:58.551327 13483 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.000s sys 0.003s
I20260927 14:53:58.554355 13500 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 14:53:58.555982 13483 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260927 14:53:58.556187 13483 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790520684602251-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790520684602251-392-0/minicluster-data/ts-2/wal
uuid: "80493bf624dc4fd7b434bd8beadf232d"
format_stamp: "Formatted at 2026-09-27 14:53:57 on dist-test-slave-xk1g"
server_key: "29b955ce5d5332f77b5a22b8dd9541ac"
server_key_iv: "d0cdd4227b03893f535ea22adce649e2"
server_key_version: "kuduclusterkey@0"
I20260927 14:53:58.556895 13483 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790520684602251-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790520684602251-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790520684602251-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 14:53:58.577509 13483 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 14:53:58.580993 13483 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 14:53:58.581238 13483 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 14:53:58.581987 13483 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 14:53:58.583055 13483 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 14:53:58.583120 13483 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 14:53:58.583174 13483 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 14:53:58.583209 13483 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 14:53:58.597416 13483 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:45701
I20260927 14:53:58.598817 13483 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790520684602251-392-0/minicluster-data/ts-2/data/info.pb
I20260927 14:53:58.600978 13613 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:45701 every 8 connection(s)
I20260927 14:53:58.606289 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6936m5/build/debug/bin/kudu as pid 13483
Sep 27 14:53:58 dist-test-slave-xk1g krb5kdc[12152](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790520838, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 14:53:58.615928 13350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:53:58.603163 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:46825 (local address 127.0.98.62:42115)
0927 14:53:58.603280 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0927 14:53:58.603284 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 14:53:58.603301 (+ 17us) server_negotiation.cc:407] Connection header received
0927 14:53:58.604100 (+ 799us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:53:58.604104 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:53:58.604168 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:53:58.604247 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:53:58.605799 (+ 1552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:53:58.606487 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:53:58.608278 (+ 1791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:53:58.608567 (+ 289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:53:58.608644 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:53:58.611513 (+ 2869us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:53:58.611517 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:53:58.611520 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 14:53:58.611546 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:53:58.613422 (+ 1876us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:53:58.614087 (+ 665us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:53:58.614091 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:53:58.614093 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:53:58.614186 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:53:58.614649 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:53:58.614653 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:53:58.614655 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:53:58.614865 (+ 210us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:53:58.614938 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0927 14:53:58.615683 (+ 745us) server_negotiation.cc:299] Negotiation successful
0927 14:53:58.615794 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260927 14:53:58.615818 13618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:53:58.606952 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38250 (local address 127.0.98.62:42115)
0927 14:53:58.607217 (+ 265us) server_negotiation.cc:206] Beginning negotiation
0927 14:53:58.607223 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 14:53:58.607387 (+ 164us) server_negotiation.cc:407] Connection header received
0927 14:53:58.607546 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:53:58.607549 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:53:58.607599 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:53:58.607691 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:53:58.608729 (+ 1038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:53:58.609302 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:53:58.610551 (+ 1249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:53:58.610936 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:53:58.611013 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:53:58.612601 (+ 1588us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:53:58.612604 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:53:58.612605 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 14:53:58.612644 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:53:58.614187 (+ 1543us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:53:58.614607 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:53:58.614612 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:53:58.614614 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:53:58.614693 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:53:58.614975 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:53:58.614978 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:53:58.614980 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:53:58.615144 (+ 164us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:53:58.615354 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0927 14:53:58.615496 (+ 142us) server_negotiation.cc:299] Negotiation successful
0927 14:53:58.615611 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":175,"thread_start_us":88,"threads_started":1}
I20260927 14:53:58.617739 13616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:53:58.602026 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42115 (local address 127.0.98.3:46825)
0927 14:53:58.602626 (+ 600us) negotiation.cc:107] Waiting for socket to connect
0927 14:53:58.602685 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0927 14:53:58.603812 (+ 1127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:53:58.604479 (+ 667us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:53:58.604489 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:53:58.604887 (+ 398us) client_negotiation.cc:189] Negotiated authn=SASL
0927 14:53:58.605633 (+ 746us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:53:58.605655 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:53:58.606639 (+ 984us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:53:58.606642 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:53:58.608145 (+ 1503us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:53:58.608157 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:53:58.608723 (+ 566us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:53:58.608727 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:53:58.608938 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:53:58.609715 (+ 777us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 14:53:58.609731 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 14:53:58.611314 (+ 1583us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 14:53:58.613568 (+ 2254us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 14:53:58.613575 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 14:53:58.613591 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0927 14:53:58.613947 (+ 356us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 14:53:58.614313 (+ 366us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 14:53:58.614316 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 14:53:58.614319 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 14:53:58.614507 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 14:53:58.615023 (+ 516us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 14:53:58.615030 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 14:53:58.615392 (+ 362us) client_negotiation.cc:769] Sending connection context
0927 14:53:58.615867 (+ 475us) client_negotiation.cc:240] Negotiation successful
0927 14:53:58.616617 (+ 750us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":353,"thread_start_us":150,"threads_started":1}
I20260927 14:53:58.618950 13614 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42115
I20260927 14:53:58.619223 13614 heartbeater.cc:464] Registering TS with master...
I20260927 14:53:58.619808 13614 heartbeater.cc:511] Master 127.0.98.62:42115 requested a full tablet report, sending...
W20260927 14:53:58.621536 13614 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:42115 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260927 14:53:58.621670 13614 heartbeater.cc:464] Registering TS with master...
I20260927 14:53:58.621842 13614 heartbeater.cc:511] Master 127.0.98.62:42115 requested a full tablet report, sending...
I20260927 14:53:58.622637 13614 heartbeater.cc:464] Registering TS with master...
I20260927 14:53:58.622798 13614 heartbeater.cc:511] Master 127.0.98.62:42115 requested a full tablet report, sending...
W20260927 14:53:58.623983 13614 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Sep 27 14:53:58 dist-test-slave-xk1g krb5kdc[12152](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790520838, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 14:53:59.090137 13346 heartbeater.cc:464] Registering TS with master...
I20260927 14:53:59.090463 13346 heartbeater.cc:511] Master 127.0.98.62:42115 requested a full tablet report, sending...
I20260927 14:53:59.091686 13148 ts_manager.cc:194] Registered new tserver with Master: 3c7df43e4dd84759ad0d79a4f1e4113f (127.0.98.1:41959)
I20260927 14:53:59.093451 13148 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:42355
Sep 27 14:53:59 dist-test-slave-xk1g krb5kdc[12152](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790520838, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260927 14:53:59.377395 13480 heartbeater.cc:464] Registering TS with master...
I20260927 14:53:59.377705 13480 heartbeater.cc:511] Master 127.0.98.62:42115 requested a full tablet report, sending...
I20260927 14:53:59.381111 13148 ts_manager.cc:194] Registered new tserver with Master: 86a219d425c84c2eb7193910d77e413b (127.0.98.2:36931)
I20260927 14:53:59.381878 13148 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:47445
Sep 27 14:53:59 dist-test-slave-xk1g krb5kdc[12152](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790520838, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260927 14:53:59.624256 13614 heartbeater.cc:464] Registering TS with master...
I20260927 14:53:59.624583 13614 heartbeater.cc:511] Master 127.0.98.62:42115 requested a full tablet report, sending...
I20260927 14:53:59.625451 13148 ts_manager.cc:194] Registered new tserver with Master: 80493bf624dc4fd7b434bd8beadf232d (127.0.98.3:45701)
I20260927 14:53:59.626231 13148 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:46825
I20260927 14:53:59.630770 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 14:53:59.665464 13636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:53:59.633237 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38262 (local address 127.0.98.62:42115)
0927 14:53:59.641282 (+ 8045us) server_negotiation.cc:206] Beginning negotiation
0927 14:53:59.641288 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 14:53:59.641306 (+ 18us) server_negotiation.cc:407] Connection header received
0927 14:53:59.641361 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:53:59.641364 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:53:59.641418 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:53:59.641521 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:53:59.642609 (+ 1088us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:53:59.643392 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:53:59.644536 (+ 1144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:53:59.645337 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:53:59.645443 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:53:59.652637 (+ 7194us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:53:59.652643 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:53:59.652645 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 14:53:59.653654 (+ 1009us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:53:59.661131 (+ 7477us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:53:59.662544 (+ 1413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:53:59.662547 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:53:59.662550 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:53:59.662710 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:53:59.663045 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:53:59.663048 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:53:59.663050 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:53:59.663713 (+ 663us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:53:59.665075 (+ 1362us) server_negotiation.cc:1035] Waiting for connection context
0927 14:53:59.665153 (+ 78us) server_negotiation.cc:299] Negotiation successful
0927 14:53:59.665284 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7941,"thread_start_us":106,"threads_started":1}
I20260927 14:53:59.670393 13148 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38262:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 14:53:59.675237 13148 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 14:53:59.694844 13644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:53:59.687755 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39824 (local address 127.0.98.1:41959)
0927 14:53:59.688557 (+ 802us) server_negotiation.cc:206] Beginning negotiation
0927 14:53:59.688566 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 14:53:59.688953 (+ 387us) server_negotiation.cc:407] Connection header received
0927 14:53:59.689292 (+ 339us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:53:59.689298 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:53:59.689528 (+ 230us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:53:59.689652 (+ 124us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 14:53:59.690176 (+ 524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:53:59.691299 (+ 1123us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:53:59.692944 (+ 1645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:53:59.693793 (+ 849us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:53:59.693863 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:53:59.694005 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0927 14:53:59.694409 (+ 404us) server_negotiation.cc:299] Negotiation successful
0927 14:53:59.694690 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":673,"thread_start_us":88,"threads_started":1}
I20260927 14:53:59.695760 13643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:53:59.687636 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:41959 (local address 127.0.0.1:39824)
0927 14:53:59.688813 (+ 1177us) negotiation.cc:107] Waiting for socket to connect
0927 14:53:59.688864 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0927 14:53:59.689100 (+ 236us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:53:59.689871 (+ 771us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:53:59.689879 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:53:59.689908 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 14:53:59.690042 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:53:59.690049 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:53:59.691423 (+ 1374us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:53:59.691426 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:53:59.692812 (+ 1386us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:53:59.692822 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:53:59.694098 (+ 1276us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:53:59.694101 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:53:59.694207 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:53:59.694243 (+ 36us) client_negotiation.cc:769] Sending connection context
0927 14:53:59.695304 (+ 1061us) client_negotiation.cc:240] Negotiation successful
0927 14:53:59.695581 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1002,"mutex_wait_us":32,"thread_start_us":96,"threads_started":1}
I20260927 14:53:59.696170 13647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:53:59.689222 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37146 (local address 127.0.98.2:36931)
0927 14:53:59.690061 (+ 839us) server_negotiation.cc:206] Beginning negotiation
0927 14:53:59.690067 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 14:53:59.690535 (+ 468us) server_negotiation.cc:407] Connection header received
0927 14:53:59.690607 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:53:59.690612 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:53:59.690776 (+ 164us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:53:59.690899 (+ 123us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 14:53:59.691258 (+ 359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:53:59.692360 (+ 1102us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:53:59.693779 (+ 1419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:53:59.695177 (+ 1398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:53:59.695278 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:53:59.695873 (+ 595us) server_negotiation.cc:1035] Waiting for connection context
0927 14:53:59.695927 (+ 54us) server_negotiation.cc:299] Negotiation successful
0927 14:53:59.696009 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":731,"thread_start_us":111,"threads_started":1}
I20260927 14:53:59.696408 13646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:53:59.688389 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:36931 (local address 127.0.0.1:37146)
0927 14:53:59.690349 (+ 1960us) negotiation.cc:107] Waiting for socket to connect
0927 14:53:59.690362 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 14:53:59.690444 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:53:59.691044 (+ 600us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:53:59.691047 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:53:59.691062 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 14:53:59.691148 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:53:59.691153 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:53:59.692489 (+ 1336us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:53:59.692493 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:53:59.693639 (+ 1146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:53:59.693650 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:53:59.695490 (+ 1840us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:53:59.695494 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:53:59.695576 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:53:59.695596 (+ 20us) client_negotiation.cc:769] Sending connection context
0927 14:53:59.695659 (+ 63us) client_negotiation.cc:240] Negotiation successful
0927 14:53:59.695728 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1906,"mutex_wait_us":1,"thread_start_us":76,"threads_started":1}
I20260927 14:53:59.699075 13406 tablet_service.cc:1467] Processing CreateTablet for tablet fd40dea5470a4d5dbdf2de30e7724e25 (DEFAULT_TABLE table=test-table [id=25adf8bc179e4217817104d2ee6b2b70]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 14:53:59.699154 13280 tablet_service.cc:1467] Processing CreateTablet for tablet fd40dea5470a4d5dbdf2de30e7724e25 (DEFAULT_TABLE table=test-table [id=25adf8bc179e4217817104d2ee6b2b70]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 14:53:59.700075 13406 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fd40dea5470a4d5dbdf2de30e7724e25. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 14:53:59.700127 13280 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fd40dea5470a4d5dbdf2de30e7724e25. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 14:53:59.708624 13650 tablet_bootstrap.cc:492] T fd40dea5470a4d5dbdf2de30e7724e25 P 3c7df43e4dd84759ad0d79a4f1e4113f: Bootstrap starting.
I20260927 14:53:59.710036 13649 tablet_bootstrap.cc:492] T fd40dea5470a4d5dbdf2de30e7724e25 P 86a219d425c84c2eb7193910d77e413b: Bootstrap starting.
I20260927 14:53:59.711318 13650 tablet_bootstrap.cc:654] T fd40dea5470a4d5dbdf2de30e7724e25 P 3c7df43e4dd84759ad0d79a4f1e4113f: Neither blocks nor log segments found. Creating new log.
I20260927 14:53:59.712227 13650 log.cc:824] T fd40dea5470a4d5dbdf2de30e7724e25 P 3c7df43e4dd84759ad0d79a4f1e4113f: Log is configured to *not* fsync() on all Append() calls
I20260927 14:53:59.712909 13646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:53:59.688679 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:45701 (local address 127.0.0.1:36440)
0927 14:53:59.696545 (+ 7866us) negotiation.cc:107] Waiting for socket to connect
0927 14:53:59.696562 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 14:53:59.697318 (+ 756us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:53:59.708322 (+ 11004us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:53:59.708325 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:53:59.708347 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 14:53:59.708447 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:53:59.708453 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:53:59.709718 (+ 1265us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:53:59.709725 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:53:59.711061 (+ 1336us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:53:59.711077 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:53:59.712357 (+ 1280us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:53:59.712361 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:53:59.712554 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:53:59.712583 (+ 29us) client_negotiation.cc:769] Sending connection context
0927 14:53:59.712665 (+ 82us) client_negotiation.cc:240] Negotiation successful
0927 14:53:59.712763 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7801,"thread_start_us":74,"threads_started":1}
I20260927 14:53:59.713449 13648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:53:59.689577 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36440 (local address 127.0.98.3:45701)
0927 14:53:59.690225 (+ 648us) server_negotiation.cc:206] Beginning negotiation
0927 14:53:59.690233 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 14:53:59.697158 (+ 6925us) server_negotiation.cc:407] Connection header received
0927 14:53:59.707720 (+ 10562us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:53:59.707728 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:53:59.707948 (+ 220us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:53:59.708127 (+ 179us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 14:53:59.708585 (+ 458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:53:59.709598 (+ 1013us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:53:59.711219 (+ 1621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:53:59.712026 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:53:59.712137 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:53:59.712290 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0927 14:53:59.713197 (+ 907us) server_negotiation.cc:299] Negotiation successful
0927 14:53:59.713307 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":554,"thread_start_us":97,"threads_started":1}
I20260927 14:53:59.716041 13650 tablet_bootstrap.cc:492] T fd40dea5470a4d5dbdf2de30e7724e25 P 3c7df43e4dd84759ad0d79a4f1e4113f: No bootstrap required, opened a new log
I20260927 14:53:59.716068 13649 tablet_bootstrap.cc:654] T fd40dea5470a4d5dbdf2de30e7724e25 P 86a219d425c84c2eb7193910d77e413b: Neither blocks nor log segments found. Creating new log.
I20260927 14:53:59.716269 13650 ts_tablet_manager.cc:1397] T fd40dea5470a4d5dbdf2de30e7724e25 P 3c7df43e4dd84759ad0d79a4f1e4113f: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.000s
I20260927 14:53:59.717167 13649 log.cc:824] T fd40dea5470a4d5dbdf2de30e7724e25 P 86a219d425c84c2eb7193910d77e413b: Log is configured to *not* fsync() on all Append() calls
I20260927 14:53:59.716686 13548 tablet_service.cc:1467] Processing CreateTablet for tablet fd40dea5470a4d5dbdf2de30e7724e25 (DEFAULT_TABLE table=test-table [id=25adf8bc179e4217817104d2ee6b2b70]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 14:53:59.718904 13548 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fd40dea5470a4d5dbdf2de30e7724e25. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 14:53:59.720765 13650 raft_consensus.cc:348] T fd40dea5470a4d5dbdf2de30e7724e25 P 3c7df43e4dd84759ad0d79a4f1e4113f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c7df43e4dd84759ad0d79a4f1e4113f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41959 } } peers { permanent_uuid: "86a219d425c84c2eb7193910d77e413b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36931 } } peers { permanent_uuid: "80493bf624dc4fd7b434bd8beadf232d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45701 } }
I20260927 14:53:59.721089 13650 raft_consensus.cc:374] T fd40dea5470a4d5dbdf2de30e7724e25 P 3c7df43e4dd84759ad0d79a4f1e4113f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 14:53:59.721146 13650 raft_consensus.cc:729] T fd40dea5470a4d5dbdf2de30e7724e25 P 3c7df43e4dd84759ad0d79a4f1e4113f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c7df43e4dd84759ad0d79a4f1e4113f, State: Initialized, Role: FOLLOWER
I20260927 14:53:59.721648 13650 consensus_queue.cc:260] T fd40dea5470a4d5dbdf2de30e7724e25 P 3c7df43e4dd84759ad0d79a4f1e4113f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c7df43e4dd84759ad0d79a4f1e4113f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41959 } } peers { permanent_uuid: "86a219d425c84c2eb7193910d77e413b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36931 } } peers { permanent_uuid: "80493bf624dc4fd7b434bd8beadf232d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45701 } }
I20260927 14:53:59.722747 13650 ts_tablet_manager.cc:1428] T fd40dea5470a4d5dbdf2de30e7724e25 P 3c7df43e4dd84759ad0d79a4f1e4113f: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260927 14:53:59.723119 13346 heartbeater.cc:503] Master 127.0.98.62:42115 was elected leader, sending a full tablet report...
I20260927 14:53:59.730335 13654 tablet_bootstrap.cc:492] T fd40dea5470a4d5dbdf2de30e7724e25 P 80493bf624dc4fd7b434bd8beadf232d: Bootstrap starting.
I20260927 14:53:59.730633 13649 tablet_bootstrap.cc:492] T fd40dea5470a4d5dbdf2de30e7724e25 P 86a219d425c84c2eb7193910d77e413b: No bootstrap required, opened a new log
I20260927 14:53:59.730875 13649 ts_tablet_manager.cc:1397] T fd40dea5470a4d5dbdf2de30e7724e25 P 86a219d425c84c2eb7193910d77e413b: Time spent bootstrapping tablet: real 0.021s user 0.004s sys 0.000s
I20260927 14:53:59.732841 13654 tablet_bootstrap.cc:654] T fd40dea5470a4d5dbdf2de30e7724e25 P 80493bf624dc4fd7b434bd8beadf232d: Neither blocks nor log segments found. Creating new log.
I20260927 14:53:59.734426 13654 log.cc:824] T fd40dea5470a4d5dbdf2de30e7724e25 P 80493bf624dc4fd7b434bd8beadf232d: Log is configured to *not* fsync() on all Append() calls
I20260927 14:53:59.735078 13649 raft_consensus.cc:348] T fd40dea5470a4d5dbdf2de30e7724e25 P 86a219d425c84c2eb7193910d77e413b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c7df43e4dd84759ad0d79a4f1e4113f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41959 } } peers { permanent_uuid: "86a219d425c84c2eb7193910d77e413b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36931 } } peers { permanent_uuid: "80493bf624dc4fd7b434bd8beadf232d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45701 } }
I20260927 14:53:59.735378 13649 raft_consensus.cc:374] T fd40dea5470a4d5dbdf2de30e7724e25 P 86a219d425c84c2eb7193910d77e413b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 14:53:59.735440 13649 raft_consensus.cc:729] T fd40dea5470a4d5dbdf2de30e7724e25 P 86a219d425c84c2eb7193910d77e413b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 86a219d425c84c2eb7193910d77e413b, State: Initialized, Role: FOLLOWER
I20260927 14:53:59.735970 13649 consensus_queue.cc:260] T fd40dea5470a4d5dbdf2de30e7724e25 P 86a219d425c84c2eb7193910d77e413b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c7df43e4dd84759ad0d79a4f1e4113f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41959 } } peers { permanent_uuid: "86a219d425c84c2eb7193910d77e413b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36931 } } peers { permanent_uuid: "80493bf624dc4fd7b434bd8beadf232d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45701 } }
I20260927 14:53:59.736879 13654 tablet_bootstrap.cc:492] T fd40dea5470a4d5dbdf2de30e7724e25 P 80493bf624dc4fd7b434bd8beadf232d: No bootstrap required, opened a new log
I20260927 14:53:59.737092 13654 ts_tablet_manager.cc:1397] T fd40dea5470a4d5dbdf2de30e7724e25 P 80493bf624dc4fd7b434bd8beadf232d: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20260927 14:53:59.737178 13649 ts_tablet_manager.cc:1428] T fd40dea5470a4d5dbdf2de30e7724e25 P 86a219d425c84c2eb7193910d77e413b: Time spent starting tablet: real 0.006s user 0.008s sys 0.000s
I20260927 14:53:59.737528 13480 heartbeater.cc:503] Master 127.0.98.62:42115 was elected leader, sending a full tablet report...
I20260927 14:53:59.741012 13654 raft_consensus.cc:348] T fd40dea5470a4d5dbdf2de30e7724e25 P 80493bf624dc4fd7b434bd8beadf232d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c7df43e4dd84759ad0d79a4f1e4113f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41959 } } peers { permanent_uuid: "86a219d425c84c2eb7193910d77e413b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36931 } } peers { permanent_uuid: "80493bf624dc4fd7b434bd8beadf232d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45701 } }
I20260927 14:53:59.741309 13654 raft_consensus.cc:374] T fd40dea5470a4d5dbdf2de30e7724e25 P 80493bf624dc4fd7b434bd8beadf232d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 14:53:59.741369 13654 raft_consensus.cc:729] T fd40dea5470a4d5dbdf2de30e7724e25 P 80493bf624dc4fd7b434bd8beadf232d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 80493bf624dc4fd7b434bd8beadf232d, State: Initialized, Role: FOLLOWER
I20260927 14:53:59.741875 13654 consensus_queue.cc:260] T fd40dea5470a4d5dbdf2de30e7724e25 P 80493bf624dc4fd7b434bd8beadf232d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c7df43e4dd84759ad0d79a4f1e4113f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41959 } } peers { permanent_uuid: "86a219d425c84c2eb7193910d77e413b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36931 } } peers { permanent_uuid: "80493bf624dc4fd7b434bd8beadf232d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45701 } }
I20260927 14:53:59.742941 13654 ts_tablet_manager.cc:1428] T fd40dea5470a4d5dbdf2de30e7724e25 P 80493bf624dc4fd7b434bd8beadf232d: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260927 14:53:59.743294 13614 heartbeater.cc:503] Master 127.0.98.62:42115 was elected leader, sending a full tablet report...
I20260927 14:53:59.791263 13657 raft_consensus.cc:482] T fd40dea5470a4d5dbdf2de30e7724e25 P 80493bf624dc4fd7b434bd8beadf232d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 14:53:59.791527 13657 raft_consensus.cc:504] T fd40dea5470a4d5dbdf2de30e7724e25 P 80493bf624dc4fd7b434bd8beadf232d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c7df43e4dd84759ad0d79a4f1e4113f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41959 } } peers { permanent_uuid: "86a219d425c84c2eb7193910d77e413b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36931 } } peers { permanent_uuid: "80493bf624dc4fd7b434bd8beadf232d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45701 } }
I20260927 14:53:59.793015 13657 leader_election.cc:288] T fd40dea5470a4d5dbdf2de30e7724e25 P 80493bf624dc4fd7b434bd8beadf232d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3c7df43e4dd84759ad0d79a4f1e4113f (127.0.98.1:41959), 86a219d425c84c2eb7193910d77e413b (127.0.98.2:36931)
I20260927 14:53:59.800513 13644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:53:59.794006 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:43557 (local address 127.0.98.1:41959)
0927 14:53:59.794243 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0927 14:53:59.794248 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 14:53:59.795035 (+ 787us) server_negotiation.cc:407] Connection header received
0927 14:53:59.795892 (+ 857us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:53:59.795897 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:53:59.795964 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:53:59.796042 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 14:53:59.796576 (+ 534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:53:59.797343 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:53:59.798983 (+ 1640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:53:59.799648 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:53:59.799747 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:53:59.799829 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0927 14:53:59.800250 (+ 421us) server_negotiation.cc:299] Negotiation successful
0927 14:53:59.800347 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":160}
I20260927 14:53:59.801021 13659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:53:59.793273 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:41959 (local address 127.0.98.3:43557)
0927 14:53:59.794916 (+ 1643us) negotiation.cc:107] Waiting for socket to connect
0927 14:53:59.794935 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 14:53:59.795132 (+ 197us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:53:59.796217 (+ 1085us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:53:59.796220 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:53:59.796242 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 14:53:59.796403 (+ 161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:53:59.796412 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:53:59.797505 (+ 1093us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:53:59.797509 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:53:59.798815 (+ 1306us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:53:59.798828 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:53:59.799961 (+ 1133us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:53:59.799965 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:53:59.800073 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:53:59.800104 (+ 31us) client_negotiation.cc:769] Sending connection context
0927 14:53:59.800764 (+ 660us) client_negotiation.cc:240] Negotiation successful
0927 14:53:59.800856 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1562,"thread_start_us":110,"threads_started":1}
I20260927 14:53:59.801816 13283 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fd40dea5470a4d5dbdf2de30e7724e25" candidate_uuid: "80493bf624dc4fd7b434bd8beadf232d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c7df43e4dd84759ad0d79a4f1e4113f" is_pre_election: true
I20260927 14:53:59.802208 13283 raft_consensus.cc:2454] T fd40dea5470a4d5dbdf2de30e7724e25 P 3c7df43e4dd84759ad0d79a4f1e4113f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 80493bf624dc4fd7b434bd8beadf232d in term 0.
I20260927 14:53:59.803697 13501 leader_election.cc:302] T fd40dea5470a4d5dbdf2de30e7724e25 P 80493bf624dc4fd7b434bd8beadf232d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c7df43e4dd84759ad0d79a4f1e4113f, 80493bf624dc4fd7b434bd8beadf232d; no voters:
I20260927 14:53:59.804109 13657 raft_consensus.cc:2781] T fd40dea5470a4d5dbdf2de30e7724e25 P 80493bf624dc4fd7b434bd8beadf232d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 14:53:59.804207 13657 raft_consensus.cc:482] T fd40dea5470a4d5dbdf2de30e7724e25 P 80493bf624dc4fd7b434bd8beadf232d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 14:53:59.804265 13657 raft_consensus.cc:3037] T fd40dea5470a4d5dbdf2de30e7724e25 P 80493bf624dc4fd7b434bd8beadf232d [term 0 FOLLOWER]: Advancing to term 1
I20260927 14:53:59.806222 13657 raft_consensus.cc:504] T fd40dea5470a4d5dbdf2de30e7724e25 P 80493bf624dc4fd7b434bd8beadf232d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c7df43e4dd84759ad0d79a4f1e4113f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41959 } } peers { permanent_uuid: "86a219d425c84c2eb7193910d77e413b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36931 } } peers { permanent_uuid: "80493bf624dc4fd7b434bd8beadf232d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45701 } }
I20260927 14:53:59.806710 13657 leader_election.cc:288] T fd40dea5470a4d5dbdf2de30e7724e25 P 80493bf624dc4fd7b434bd8beadf232d [CANDIDATE]: Term 1 election: Requested vote from peers 3c7df43e4dd84759ad0d79a4f1e4113f (127.0.98.1:41959), 86a219d425c84c2eb7193910d77e413b (127.0.98.2:36931)
I20260927 14:53:59.807359 13283 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fd40dea5470a4d5dbdf2de30e7724e25" candidate_uuid: "80493bf624dc4fd7b434bd8beadf232d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c7df43e4dd84759ad0d79a4f1e4113f"
I20260927 14:53:59.807542 13283 raft_consensus.cc:3037] T fd40dea5470a4d5dbdf2de30e7724e25 P 3c7df43e4dd84759ad0d79a4f1e4113f [term 0 FOLLOWER]: Advancing to term 1
I20260927 14:53:59.809005 13283 raft_consensus.cc:2454] T fd40dea5470a4d5dbdf2de30e7724e25 P 3c7df43e4dd84759ad0d79a4f1e4113f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 80493bf624dc4fd7b434bd8beadf232d in term 1.
I20260927 14:53:59.809399 13501 leader_election.cc:302] T fd40dea5470a4d5dbdf2de30e7724e25 P 80493bf624dc4fd7b434bd8beadf232d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c7df43e4dd84759ad0d79a4f1e4113f, 80493bf624dc4fd7b434bd8beadf232d; no voters:
I20260927 14:53:59.809633 13657 raft_consensus.cc:2781] T fd40dea5470a4d5dbdf2de30e7724e25 P 80493bf624dc4fd7b434bd8beadf232d [term 1 FOLLOWER]: Leader election won for term 1
I20260927 14:53:59.809880 13657 raft_consensus.cc:686] T fd40dea5470a4d5dbdf2de30e7724e25 P 80493bf624dc4fd7b434bd8beadf232d [term 1 LEADER]: Becoming Leader. State: Replica: 80493bf624dc4fd7b434bd8beadf232d, State: Running, Role: LEADER
I20260927 14:53:59.810322 13657 consensus_queue.cc:237] T fd40dea5470a4d5dbdf2de30e7724e25 P 80493bf624dc4fd7b434bd8beadf232d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c7df43e4dd84759ad0d79a4f1e4113f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41959 } } peers { permanent_uuid: "86a219d425c84c2eb7193910d77e413b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36931 } } peers { permanent_uuid: "80493bf624dc4fd7b434bd8beadf232d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45701 } }
W20260927 14:53:59.811738 13347 tablet.cc:2369] T fd40dea5470a4d5dbdf2de30e7724e25 P 3c7df43e4dd84759ad0d79a4f1e4113f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 14:53:59.814405 13647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:53:59.794125 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:39969 (local address 127.0.98.2:36931)
0927 14:53:59.794324 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0927 14:53:59.794328 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 14:53:59.797354 (+ 3026us) server_negotiation.cc:407] Connection header received
0927 14:53:59.809357 (+ 12003us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:53:59.809362 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:53:59.809430 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:53:59.809494 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 14:53:59.810031 (+ 537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:53:59.810752 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:53:59.812971 (+ 2219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:53:59.813606 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:53:59.813697 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:53:59.813769 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0927 14:53:59.814180 (+ 411us) server_negotiation.cc:299] Negotiation successful
0927 14:53:59.814268 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":139}
I20260927 14:53:59.814033 13147 catalog_manager.cc:5495] T fd40dea5470a4d5dbdf2de30e7724e25 P 80493bf624dc4fd7b434bd8beadf232d reported cstate change: term changed from 0 to 1, leader changed from <none> to 80493bf624dc4fd7b434bd8beadf232d (127.0.98.3). New cstate: current_term: 1 leader_uuid: "80493bf624dc4fd7b434bd8beadf232d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c7df43e4dd84759ad0d79a4f1e4113f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41959 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "86a219d425c84c2eb7193910d77e413b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36931 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "80493bf624dc4fd7b434bd8beadf232d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45701 } health_report { overall_health: HEALTHY } } }
I20260927 14:53:59.814867 13658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:53:59.793623 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:36931 (local address 127.0.98.3:39969)
0927 14:53:59.797251 (+ 3628us) negotiation.cc:107] Waiting for socket to connect
0927 14:53:59.797267 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 14:53:59.809160 (+ 11893us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:53:59.809670 (+ 510us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:53:59.809674 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:53:59.809697 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 14:53:59.809850 (+ 153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:53:59.809861 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:53:59.810916 (+ 1055us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:53:59.810919 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:53:59.812805 (+ 1886us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:53:59.812816 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:53:59.813880 (+ 1064us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:53:59.813886 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:53:59.814014 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:53:59.814042 (+ 28us) client_negotiation.cc:769] Sending connection context
0927 14:53:59.814629 (+ 587us) client_negotiation.cc:240] Negotiation successful
0927 14:53:59.814717 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3561,"thread_start_us":87,"threads_started":1}
I20260927 14:53:59.815776 13422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fd40dea5470a4d5dbdf2de30e7724e25" candidate_uuid: "80493bf624dc4fd7b434bd8beadf232d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "86a219d425c84c2eb7193910d77e413b"
I20260927 14:53:59.816033 13422 raft_consensus.cc:3037] T fd40dea5470a4d5dbdf2de30e7724e25 P 86a219d425c84c2eb7193910d77e413b [term 0 FOLLOWER]: Advancing to term 1
I20260927 14:53:59.816263 13420 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fd40dea5470a4d5dbdf2de30e7724e25" candidate_uuid: "80493bf624dc4fd7b434bd8beadf232d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "86a219d425c84c2eb7193910d77e413b" is_pre_election: true
I20260927 14:53:59.817512 13422 raft_consensus.cc:2454] T fd40dea5470a4d5dbdf2de30e7724e25 P 86a219d425c84c2eb7193910d77e413b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 80493bf624dc4fd7b434bd8beadf232d in term 1.
W20260927 14:53:59.827261 13481 tablet.cc:2369] T fd40dea5470a4d5dbdf2de30e7724e25 P 86a219d425c84c2eb7193910d77e413b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 14:53:59.840368 13648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:53:59.835211 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36450 (local address 127.0.98.3:45701)
0927 14:53:59.835337 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0927 14:53:59.835342 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 14:53:59.835356 (+ 14us) server_negotiation.cc:407] Connection header received
0927 14:53:59.835445 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:53:59.835449 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:53:59.835521 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:53:59.835645 (+ 124us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 14:53:59.836050 (+ 405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:53:59.836843 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:53:59.838129 (+ 1286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:53:59.838469 (+ 340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:53:59.838588 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:53:59.839022 (+ 434us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 14:53:59.839609 (+ 587us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 14:53:59.839730 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0927 14:53:59.840162 (+ 432us) server_negotiation.cc:299] Negotiation successful
0927 14:53:59.840225 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260927 14:53:59.848083 13422 raft_consensus.cc:1261] T fd40dea5470a4d5dbdf2de30e7724e25 P 86a219d425c84c2eb7193910d77e413b [term 1 FOLLOWER]: Refusing update from remote peer 80493bf624dc4fd7b434bd8beadf232d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 14:53:59.848866 13660 consensus_queue.cc:1035] T fd40dea5470a4d5dbdf2de30e7724e25 P 80493bf624dc4fd7b434bd8beadf232d [LEADER]: Connected to new peer: Peer: permanent_uuid: "86a219d425c84c2eb7193910d77e413b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36931 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 14:53:59.849257 13283 raft_consensus.cc:1261] T fd40dea5470a4d5dbdf2de30e7724e25 P 3c7df43e4dd84759ad0d79a4f1e4113f [term 1 FOLLOWER]: Refusing update from remote peer 80493bf624dc4fd7b434bd8beadf232d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 14:53:59.849880 13660 consensus_queue.cc:1035] T fd40dea5470a4d5dbdf2de30e7724e25 P 80493bf624dc4fd7b434bd8beadf232d [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c7df43e4dd84759ad0d79a4f1e4113f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41959 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260927 14:53:59.853595 13615 tablet.cc:2369] T fd40dea5470a4d5dbdf2de30e7724e25 P 80493bf624dc4fd7b434bd8beadf232d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 14:53:59.866036 13663 mvcc.cc:205] Tried to move back new op lower bound from 7333973360007929856 to 7333973359866900480. Current Snapshot: MvccSnapshot[applied={T|T < 7333973360007929856}]
I20260927 14:53:59.895025 13147 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38262:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 14:53:59.895505 13147 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38262:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 14:53:59.898556 13147 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ae16374104dc4823a1362be92f41ed52: Sending DeleteTablet for 3 replicas of tablet fd40dea5470a4d5dbdf2de30e7724e25
I20260927 14:53:59.900115 13406 tablet_service.cc:1514] Processing DeleteTablet for tablet fd40dea5470a4d5dbdf2de30e7724e25 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 14:53:59 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:37146
I20260927 14:53:59.900118 13548 tablet_service.cc:1514] Processing DeleteTablet for tablet fd40dea5470a4d5dbdf2de30e7724e25 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 14:53:59 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:36440
I20260927 14:53:59.900251 13280 tablet_service.cc:1514] Processing DeleteTablet for tablet fd40dea5470a4d5dbdf2de30e7724e25 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 14:53:59 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:39824
I20260927 14:53:59.900692 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6936m5/build/debug/bin/kudu with pid 13186
I20260927 14:53:59.910919 13672 tablet_replica.cc:330] T fd40dea5470a4d5dbdf2de30e7724e25 P 86a219d425c84c2eb7193910d77e413b: stopping tablet replica
I20260927 14:53:59.911432 13672 raft_consensus.cc:2229] T fd40dea5470a4d5dbdf2de30e7724e25 P 86a219d425c84c2eb7193910d77e413b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 14:53:59.911927 13672 raft_consensus.cc:2258] T fd40dea5470a4d5dbdf2de30e7724e25 P 86a219d425c84c2eb7193910d77e413b [term 1 FOLLOWER]: Raft consensus is shut down!
W20260927 14:53:59.914417 13132 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:41959 (error 108)
I20260927 14:53:59.915290 13673 tablet_replica.cc:330] T fd40dea5470a4d5dbdf2de30e7724e25 P 80493bf624dc4fd7b434bd8beadf232d: stopping tablet replica
I20260927 14:53:59.915571 13672 ts_tablet_manager.cc:1905] T fd40dea5470a4d5dbdf2de30e7724e25 P 86a219d425c84c2eb7193910d77e413b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 14:53:59.915628 13673 raft_consensus.cc:2229] T fd40dea5470a4d5dbdf2de30e7724e25 P 80493bf624dc4fd7b434bd8beadf232d [term 1 LEADER]: Raft consensus shutting down.
I20260927 14:53:59.915916 13646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:53:59.915557 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:41959 (local address 127.0.0.1:39832)
0927 14:53:59.915756 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0927 14:53:59.915833 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:41959: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120}
I20260927 14:53:59.915985 13673 raft_consensus.cc:2258] T fd40dea5470a4d5dbdf2de30e7724e25 P 80493bf624dc4fd7b434bd8beadf232d [term 1 FOLLOWER]: Raft consensus is shut down!
W20260927 14:53:59.916090 13132 catalog_manager.cc:4584] TS 3c7df43e4dd84759ad0d79a4f1e4113f (127.0.98.1:41959): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet fd40dea5470a4d5dbdf2de30e7724e25: Network error: Client connection negotiation failed: client connection to 127.0.98.1:41959: connect: Connection refused (error 111)
I20260927 14:53:59.916405 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6936m5/build/debug/bin/kudu with pid 13349
I20260927 14:53:59.922906 13673 ts_tablet_manager.cc:1905] T fd40dea5470a4d5dbdf2de30e7724e25 P 80493bf624dc4fd7b434bd8beadf232d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 14:53:59.923326 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6936m5/build/debug/bin/kudu with pid 13483
I20260927 14:53:59.933604 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6936m5/build/debug/bin/kudu with pid 13115
I20260927 14:53:59.943625 392 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260927 14:54:00.278386 392 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260927 14:54:00.278486 392 mini_ranger.cc:66] Stopping Ranger...
I20260927 14:54:00.625202 392 mini_ranger.cc:68] Stopped Ranger
2026-09-27 14:54:00.625 UTC [12182] LOG: received smart shutdown request
2026-09-27 14:54:00.629 UTC [12182] LOG: background worker "logical replication launcher" (PID 12191) exited with exit code 1
2026-09-27 14:54:00.629 UTC [12186] LOG: shutting down
2026-09-27 14:54:00.807 UTC [12182] LOG: database system is shut down
2026-09-27T14:54:00Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (47538 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790520684602251-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 14:54:00 dist-test-slave-xk1g krb5kdc[13684](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 14:54:00 dist-test-slave-xk1g krb5kdc[13684](info): set up 2 sockets
Sep 27 14:54:00 dist-test-slave-xk1g krb5kdc[13684](info): commencing operation
krb5kdc: starting...
W20260927 14:54:01.965211 392 mini_kdc.cc:121] Time spent starting KDC: real 1.023s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790520684602251-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790520684602251-392-0/krb5kdc/test-user.keytab.
Sep 27 14:54:02 dist-test-slave-xk1g krb5kdc[13684](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790520842, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T14:54:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T14:54:02Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790520684602251-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790520684602251-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 14:54:02.092003 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6936m5/build/debug/bin/kudu
/tmp/dist-test-task6936m5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790520684602251-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790520684602251-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790520684602251-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790520684602251-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:32875
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790520684602251-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34277
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:32875,127.0.98.61:41887,127.0.98.60:34009
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790520684602251-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790520684602251-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790520684602251-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 14:54:02.203763 13697 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 14:54:02.204053 13697 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 14:54:02.204136 13697 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 14:54:02.208235 13697 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 14:54:02.208325 13697 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 14:54:02.208360 13697 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 14:54:02.208395 13697 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 14:54:02.208431 13697 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 14:54:02.213662 13697 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34277
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790520684602251-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790520684602251-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:32875,127.0.98.61:41887,127.0.98.60:34009
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790520684602251-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:32875
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790520684602251-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13697
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790520684602251-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:15:17 UTC on ccycloud.quasar-ledxfe.root.comops.site
build id 4676134
I20260927 14:54:02.214757 13697 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 14:54:02.215768 13697 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 14:54:02.224094 13703 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 14:54:02.224164 13705 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 14:54:02.224172 13702 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 14:54:02.224251 13697 server_base.cc:1034] running on GCE node
I20260927 14:54:02.225174 13697 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 14:54:02.226240 13697 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 14:54:02.227447 13697 hybrid_clock.cc:648] HybridClock initialized: now 1790520842227418 us; error 39 us; skew 500 ppm
Sep 27 14:54:02 dist-test-slave-xk1g krb5kdc[13684](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790520842, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 14:54:02.230265 13697 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 14:54:02.231088 13697 webserver.cc:460] Webserver started at http://127.0.98.62:33775/ using document root <none> and password file <none>
I20260927 14:54:02.231585 13697 fs_manager.cc:359] Metadata directory not provided
I20260927 14:54:02.231640 13697 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 14:54:02.231794 13697 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 14:54:02.233814 13697 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790520684602251-392-0/minicluster-data/master-0/data/instance:
uuid: "a1db0dcf3e694425a6dd5f84316ad738"
format_stamp: "Formatted at 2026-09-27 14:54:02 on dist-test-slave-xk1g"
server_key: "7a68279911d51829ac1f56f9eef4309c"
server_key_iv: "3b16fd71eb59f04ba54747f6331529ba"
server_key_version: "encryptionkey@0"
I20260927 14:54:02.234334 13697 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790520684602251-392-0/minicluster-data/master-0/wal/instance:
uuid: "a1db0dcf3e694425a6dd5f84316ad738"
format_stamp: "Formatted at 2026-09-27 14:54:02 on dist-test-slave-xk1g"
server_key: "7a68279911d51829ac1f56f9eef4309c"
server_key_iv: "3b16fd71eb59f04ba54747f6331529ba"
server_key_version: "encryptionkey@0"
I20260927 14:54:02.237989 13697 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 14:54:02.240325 13712 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 14:54:02.241619 13697 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 14:54:02.241760 13697 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790520684602251-392-0/minicluster-data/master-0/data,/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790520684602251-392-0/minicluster-data/master-0/wal
uuid: "a1db0dcf3e694425a6dd5f84316ad738"
format_stamp: "Formatted at 2026-09-27 14:54:02 on dist-test-slave-xk1g"
server_key: "7a68279911d51829ac1f56f9eef4309c"
server_key_iv: "3b16fd71eb59f04ba54747f6331529ba"
server_key_version: "encryptionkey@0"
I20260927 14:54:02.241864 13697 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790520684602251-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790520684602251-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790520684602251-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 14:54:02.259754 13697 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 14:54:02.262801 13697 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 14:54:02.263007 13697 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 14:54:02.270187 13697 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:32875
I20260927 14:54:02.270207 13764 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:32875 every 8 connection(s)
I20260927 14:54:02.271167 13697 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790520684602251-392-0/minicluster-data/master-0/data/info.pb
I20260927 14:54:02.273901 13765 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 14:54:02.278108 13765 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32875 } has no permanent_uuid. Determining permanent_uuid...
I20260927 14:54:02.278908 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6936m5/build/debug/bin/kudu as pid 13697
I20260927 14:54:02.279058 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790520684602251-392-0/minicluster-data/master-0/wal/instance
I20260927 14:54:02.279335 392 external_mini_cluster.cc:1428] Setting key 50420db33bff320386357cd3c4de1ab6
WARNING: no policy specified for kudu/127.0.98.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.61@KRBTEST.COM" created.
Sep 27 14:54:02 dist-test-slave-xk1g krb5kdc[13684](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790520842, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 14:54:02.295594 13768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:54:02.280442 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:32875 (local address 127.0.0.1:50830)
0927 14:54:02.280985 (+ 543us) negotiation.cc:107] Waiting for socket to connect
0927 14:54:02.281028 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0927 14:54:02.281975 (+ 947us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:54:02.282653 (+ 678us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:54:02.282659 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:54:02.283111 (+ 452us) client_negotiation.cc:189] Negotiated authn=SASL
0927 14:54:02.283346 (+ 235us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:54:02.283359 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:54:02.284508 (+ 1149us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:54:02.284510 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:54:02.285311 (+ 801us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:54:02.285320 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:54:02.285947 (+ 627us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:54:02.285951 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:54:02.286038 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:54:02.286730 (+ 692us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 14:54:02.286758 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0927 14:54:02.290060 (+ 3302us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 14:54:02.292286 (+ 2226us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 14:54:02.292299 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 14:54:02.292313 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0927 14:54:02.292638 (+ 325us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 14:54:02.292995 (+ 357us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 14:54:02.293001 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 14:54:02.293006 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 14:54:02.293185 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 14:54:02.293833 (+ 648us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 14:54:02.293839 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 14:54:02.293989 (+ 150us) client_negotiation.cc:769] Sending connection context
0927 14:54:02.294161 (+ 172us) client_negotiation.cc:240] Negotiation successful
0927 14:54:02.294559 (+ 398us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":353,"mutex_wait_us":114,"spinlock_wait_cycles":2816,"thread_start_us":171,"threads_started":1}
I20260927 14:54:02.295601 13769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:54:02.281462 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50830 (local address 127.0.98.62:32875)
0927 14:54:02.281749 (+ 287us) server_negotiation.cc:206] Beginning negotiation
0927 14:54:02.281760 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 14:54:02.281804 (+ 44us) server_negotiation.cc:407] Connection header received
0927 14:54:02.282284 (+ 480us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:54:02.282288 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:54:02.282467 (+ 179us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:54:02.282662 (+ 195us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:54:02.283492 (+ 830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:54:02.284359 (+ 867us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:54:02.285424 (+ 1065us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:54:02.285799 (+ 375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:54:02.285968 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:54:02.290229 (+ 4261us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:54:02.290241 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:54:02.290252 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 14:54:02.290273 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:54:02.292168 (+ 1895us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:54:02.292775 (+ 607us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:54:02.292781 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:54:02.292782 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:54:02.292891 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:54:02.293306 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:54:02.293308 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:54:02.293309 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:54:02.293703 (+ 394us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:54:02.293865 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0927 14:54:02.294240 (+ 375us) server_negotiation.cc:299] Negotiation successful
0927 14:54:02.294505 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":190,"thread_start_us":88,"threads_started":1}
I20260927 14:54:02.298394 13765 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41887 } has no permanent_uuid. Determining permanent_uuid...
I20260927 14:54:02.299212 13768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:54:02.298888 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:41887 (local address 127.0.0.1:45960)
0927 14:54:02.299058 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0927 14:54:02.299143 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:41887: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
W20260927 14:54:02.299475 13713 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.61:41887: connect: Connection refused (error 111)
I20260927 14:54:02.300341 13768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:54:02.300135 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:41887 (local address 127.0.0.1:45964)
0927 14:54:02.300244 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0927 14:54:02.300287 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:41887: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260927 14:54:02.300542 13765 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:41887: Network error: Client connection negotiation failed: client connection to 127.0.98.61:41887: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790520684602251-392-0/krb5kdc/kudu_127.0.98.61.keytab.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790520684602251-392-0/krb5kdc/kudu_127.0.98.61.keytab.
I20260927 14:54:02.307909 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6936m5/build/debug/bin/kudu
/tmp/dist-test-task6936m5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790520684602251-392-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790520684602251-392-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790520684602251-392-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790520684602251-392-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:41887
--webserver_interface=127.0.98.61
--webserver_port=0
--keytab_file=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790520684602251-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34277
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:32875,127.0.98.61:41887,127.0.98.60:34009
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790520684602251-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790520684602251-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790520684602251-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 14:54:02.349800 13765 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41887 } attempt: 1
I20260927 14:54:02.350775 13768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:54:02.350468 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:41887 (local address 127.0.0.1:45966)
0927 14:54:02.350631 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0927 14:54:02.350694 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:41887: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
I20260927 14:54:02.351521 13768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:54:02.351332 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:41887 (local address 127.0.0.1:45972)
0927 14:54:02.351425 (+ 93us) negotiation.cc:107] Waiting for socket to connect
0927 14:54:02.351469 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:41887: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260927 14:54:02.351662 13765 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:41887: Network error: Client connection negotiation failed: client connection to 127.0.98.61:41887: connect: Connection refused (error 111)
W20260927 14:54:02.418113 13772 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 14:54:02.418357 13772 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 14:54:02.418396 13772 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260927 14:54:02.419835 13765 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41887 } attempt: 2
I20260927 14:54:02.420773 13768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:54:02.420449 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:41887 (local address 127.0.0.1:45978)
0927 14:54:02.420633 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0927 14:54:02.420697 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:41887: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
I20260927 14:54:02.421566 13768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:54:02.421373 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:41887 (local address 127.0.0.1:45982)
0927 14:54:02.421465 (+ 92us) negotiation.cc:107] Waiting for socket to connect
0927 14:54:02.421509 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:41887: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260927 14:54:02.421770 13765 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:41887: Network error: Client connection negotiation failed: client connection to 127.0.98.61:41887: connect: Connection refused (error 111)
W20260927 14:54:02.422883 13772 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 14:54:02.422937 13772 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 14:54:02.422957 13772 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 14:54:02.422994 13772 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 14:54:02.423030 13772 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 14:54:02.428260 13772 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34277
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790520684602251-392-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790520684602251-392-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:32875,127.0.98.61:41887,127.0.98.60:34009
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790520684602251-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:41887
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790520684602251-392-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.98.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13772
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790520684602251-392-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:15:17 UTC on ccycloud.quasar-ledxfe.root.comops.site
build id 4676134
I20260927 14:54:02.429348 13772 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 14:54:02.430334 13772 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 14:54:02.438858 13778 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 14:54:02.438943 13780 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 14:54:02.438970 13772 server_base.cc:1034] running on GCE node
W20260927 14:54:02.438858 13777 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 14:54:02.439692 13772 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 14:54:02.440821 13772 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 14:54:02.442010 13772 hybrid_clock.cc:648] HybridClock initialized: now 1790520842441991 us; error 30 us; skew 500 ppm
Sep 27 14:54:02 dist-test-slave-xk1g krb5kdc[13684](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790520842, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 14:54:02.444797 13772 init.cc:377] Logged in from keytab as kudu/127.0.98.61@KRBTEST.COM (short username kudu)
I20260927 14:54:02.445734 13772 webserver.cc:460] Webserver started at http://127.0.98.61:38693/ using document root <none> and password file <none>
I20260927 14:54:02.446305 13772 fs_manager.cc:359] Metadata directory not provided
I20260927 14:54:02.446367 13772 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 14:54:02.446527 13772 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 14:54:02.448459 13772 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790520684602251-392-0/minicluster-data/master-1/data/instance:
uuid: "526ee58668d6401e907cc8d97793c839"
format_stamp: "Formatted at 2026-09-27 14:54:02 on dist-test-slave-xk1g"
server_key: "b634dd07ed73890e913b12eb18e3215c"
server_key_iv: "abcfc461bec1ae4b68121a91874e5f99"
server_key_version: "encryptionkey@0"
I20260927 14:54:02.448961 13772 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790520684602251-392-0/minicluster-data/master-1/wal/instance:
uuid: "526ee58668d6401e907cc8d97793c839"
format_stamp: "Formatted at 2026-09-27 14:54:02 on dist-test-slave-xk1g"
server_key: "b634dd07ed73890e913b12eb18e3215c"
server_key_iv: "abcfc461bec1ae4b68121a91874e5f99"
server_key_version: "encryptionkey@0"
I20260927 14:54:02.452527 13772 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.002s
I20260927 14:54:02.454944 13787 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 14:54:02.456146 13772 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.001s
I20260927 14:54:02.456247 13772 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790520684602251-392-0/minicluster-data/master-1/data,/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790520684602251-392-0/minicluster-data/master-1/wal
uuid: "526ee58668d6401e907cc8d97793c839"
format_stamp: "Formatted at 2026-09-27 14:54:02 on dist-test-slave-xk1g"
server_key: "b634dd07ed73890e913b12eb18e3215c"
server_key_iv: "abcfc461bec1ae4b68121a91874e5f99"
server_key_version: "encryptionkey@0"
I20260927 14:54:02.456333 13772 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790520684602251-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790520684602251-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790520684602251-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 14:54:02.470362 13772 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 14:54:02.473335 13772 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 14:54:02.473533 13772 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 14:54:02.480695 13772 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:41887
I20260927 14:54:02.480716 13839 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:41887 every 8 connection(s)
I20260927 14:54:02.481751 13772 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790520684602251-392-0/minicluster-data/master-1/data/info.pb
I20260927 14:54:02.484054 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6936m5/build/debug/bin/kudu as pid 13772
I20260927 14:54:02.484170 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790520684602251-392-0/minicluster-data/master-1/wal/instance
I20260927 14:54:02.484493 392 external_mini_cluster.cc:1428] Setting key 9c1ef72dc759a324bb1138c132c90b76
I20260927 14:54:02.484705 13840 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 14:54:02.488845 13840 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32875 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.0.98.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.60@KRBTEST.COM" created.
Sep 27 14:54:02 dist-test-slave-xk1g krb5kdc[13684](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790520842, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 14:54:02.503894 13769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:54:02.491002 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50840 (local address 127.0.98.62:32875)
0927 14:54:02.491157 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0927 14:54:02.491162 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 14:54:02.492249 (+ 1087us) server_negotiation.cc:407] Connection header received
0927 14:54:02.493348 (+ 1099us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:54:02.493352 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:54:02.493420 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:54:02.493512 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:54:02.494772 (+ 1260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:54:02.495420 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:54:02.496644 (+ 1224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:54:02.496888 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:54:02.497017 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:54:02.499909 (+ 2892us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:54:02.499915 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:54:02.499922 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 14:54:02.499957 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:54:02.501671 (+ 1714us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:54:02.502262 (+ 591us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:54:02.502268 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:54:02.502273 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:54:02.502359 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:54:02.502689 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:54:02.502694 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:54:02.502695 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:54:02.502966 (+ 271us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:54:02.503080 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0927 14:54:02.503586 (+ 506us) server_negotiation.cc:299] Negotiation successful
0927 14:54:02.503741 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260927 14:54:02.504668 13843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:54:02.491453 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:32875 (local address 127.0.0.1:50840)
0927 14:54:02.492072 (+ 619us) negotiation.cc:107] Waiting for socket to connect
0927 14:54:02.492117 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0927 14:54:02.493039 (+ 922us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:54:02.493694 (+ 655us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:54:02.493701 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:54:02.494049 (+ 348us) client_negotiation.cc:189] Negotiated authn=SASL
0927 14:54:02.494632 (+ 583us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:54:02.494647 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:54:02.495587 (+ 940us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:54:02.495590 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:54:02.496475 (+ 885us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:54:02.496484 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:54:02.497086 (+ 602us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:54:02.497092 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:54:02.497267 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:54:02.497887 (+ 620us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 14:54:02.497912 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0927 14:54:02.499727 (+ 1815us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 14:54:02.501792 (+ 2065us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 14:54:02.501797 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 14:54:02.501809 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0927 14:54:02.502156 (+ 347us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 14:54:02.502456 (+ 300us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 14:54:02.502458 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 14:54:02.502459 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 14:54:02.502592 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 14:54:02.503081 (+ 489us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 14:54:02.503086 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 14:54:02.503330 (+ 244us) client_negotiation.cc:769] Sending connection context
0927 14:54:02.503485 (+ 155us) client_negotiation.cc:240] Negotiation successful
0927 14:54:02.503754 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":385,"thread_start_us":185,"threads_started":1}
I20260927 14:54:02.506438 13840 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41887 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 14:54:02 dist-test-slave-xk1g krb5kdc[13684](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790520842, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790520684602251-392-0/krb5kdc/kudu_127.0.98.60.keytab.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790520684602251-392-0/krb5kdc/kudu_127.0.98.60.keytab.
I20260927 14:54:02.512925 13765 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41887 } attempt: 3
I20260927 14:54:02.513291 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6936m5/build/debug/bin/kudu
/tmp/dist-test-task6936m5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790520684602251-392-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790520684602251-392-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790520684602251-392-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790520684602251-392-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:34009
--webserver_interface=127.0.98.60
--webserver_port=0
--keytab_file=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790520684602251-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34277
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:32875,127.0.98.61:41887,127.0.98.60:34009
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790520684602251-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790520684602251-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790520684602251-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 14:54:02.516566 13843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:54:02.506947 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:41887 (local address 127.0.0.1:45996)
0927 14:54:02.507065 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0927 14:54:02.507080 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 14:54:02.507150 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:54:02.507878 (+ 728us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:54:02.507881 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:54:02.508045 (+ 164us) client_negotiation.cc:189] Negotiated authn=SASL
0927 14:54:02.508162 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:54:02.508167 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:54:02.509265 (+ 1098us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:54:02.509268 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:54:02.509818 (+ 550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:54:02.509826 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:54:02.510480 (+ 654us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:54:02.510483 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:54:02.510574 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:54:02.511076 (+ 502us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 14:54:02.511088 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 14:54:02.512337 (+ 1249us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 14:54:02.514736 (+ 2399us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 14:54:02.514741 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 14:54:02.514744 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 14:54:02.514983 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 14:54:02.515279 (+ 296us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 14:54:02.515284 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 14:54:02.515289 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 14:54:02.515366 (+ 77us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 14:54:02.515892 (+ 526us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 14:54:02.515899 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 14:54:02.516012 (+ 113us) client_negotiation.cc:769] Sending connection context
0927 14:54:02.516131 (+ 119us) client_negotiation.cc:240] Negotiation successful
0927 14:54:02.516295 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":42}
I20260927 14:54:02.516902 13845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:54:02.507127 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45996 (local address 127.0.98.61:41887)
0927 14:54:02.507445 (+ 318us) server_negotiation.cc:206] Beginning negotiation
0927 14:54:02.507458 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 14:54:02.507482 (+ 24us) server_negotiation.cc:407] Connection header received
0927 14:54:02.507521 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:54:02.507526 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:54:02.507756 (+ 230us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:54:02.507902 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:54:02.508268 (+ 366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:54:02.509151 (+ 883us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:54:02.509954 (+ 803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:54:02.510337 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:54:02.510456 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:54:02.512505 (+ 2049us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:54:02.512514 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:54:02.512526 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 14:54:02.512579 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:54:02.514600 (+ 2021us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:54:02.515098 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:54:02.515105 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:54:02.515109 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:54:02.515202 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:54:02.515463 (+ 261us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:54:02.515467 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:54:02.515468 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:54:02.515779 (+ 311us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:54:02.515915 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0927 14:54:02.516235 (+ 320us) server_negotiation.cc:299] Negotiation successful
0927 14:54:02.516380 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":208,"thread_start_us":112,"threads_started":1}
I20260927 14:54:02.519258 13840 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34009 } has no permanent_uuid. Determining permanent_uuid...
I20260927 14:54:02.520067 13843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:54:02.519739 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:34009 (local address 127.0.0.1:49530)
0927 14:54:02.519883 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0927 14:54:02.519974 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:34009: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
Sep 27 14:54:02 dist-test-slave-xk1g krb5kdc[13684](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790520842, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
W20260927 14:54:02.520267 13788 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.60:34009: connect: Connection refused (error 111)
I20260927 14:54:02.521776 13843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:54:02.520942 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:34009 (local address 127.0.0.1:49538)
0927 14:54:02.521372 (+ 430us) negotiation.cc:107] Waiting for socket to connect
0927 14:54:02.521436 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:34009: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":342}
W20260927 14:54:02.521907 13840 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:34009: Network error: Client connection negotiation failed: client connection to 127.0.98.60:34009: connect: Connection refused (error 111)
I20260927 14:54:02.523906 13768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:54:02.513419 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:41887 (local address 127.0.0.1:46000)
0927 14:54:02.513821 (+ 402us) negotiation.cc:107] Waiting for socket to connect
0927 14:54:02.513839 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 14:54:02.513924 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:54:02.515211 (+ 1287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:54:02.515216 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:54:02.515409 (+ 193us) client_negotiation.cc:189] Negotiated authn=SASL
0927 14:54:02.515552 (+ 143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:54:02.515560 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:54:02.516509 (+ 949us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:54:02.516513 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:54:02.517811 (+ 1298us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:54:02.517821 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:54:02.518434 (+ 613us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:54:02.518438 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:54:02.518540 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:54:02.519051 (+ 511us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 14:54:02.519062 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 14:54:02.520508 (+ 1446us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 14:54:02.522413 (+ 1905us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 14:54:02.522417 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 14:54:02.522419 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 14:54:02.522645 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 14:54:02.522976 (+ 331us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 14:54:02.522979 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 14:54:02.522980 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 14:54:02.523054 (+ 74us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 14:54:02.523516 (+ 462us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 14:54:02.523518 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 14:54:02.523596 (+ 78us) client_negotiation.cc:769] Sending connection context
0927 14:54:02.523686 (+ 90us) client_negotiation.cc:240] Negotiation successful
0927 14:54:02.523796 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":305}
I20260927 14:54:02.524155 13846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:54:02.513491 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46000 (local address 127.0.98.61:41887)
0927 14:54:02.514904 (+ 1413us) server_negotiation.cc:206] Beginning negotiation
0927 14:54:02.514911 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 14:54:02.514923 (+ 12us) server_negotiation.cc:407] Connection header received
0927 14:54:02.514952 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:54:02.514954 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:54:02.514986 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:54:02.515157 (+ 171us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:54:02.515679 (+ 522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:54:02.516371 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:54:02.517967 (+ 1596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:54:02.518304 (+ 337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:54:02.518751 (+ 447us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:54:02.520691 (+ 1940us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:54:02.520695 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:54:02.520699 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 14:54:02.520722 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:54:02.522302 (+ 1580us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:54:02.522760 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:54:02.522765 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:54:02.522770 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:54:02.522869 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:54:02.523147 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:54:02.523151 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:54:02.523154 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:54:02.523409 (+ 255us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:54:02.523522 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0927 14:54:02.523791 (+ 269us) server_negotiation.cc:299] Negotiation successful
0927 14:54:02.523920 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1339,"thread_start_us":122,"threads_started":1}
I20260927 14:54:02.524819 13765 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34009 } has no permanent_uuid. Determining permanent_uuid...
I20260927 14:54:02.525552 13768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:54:02.525340 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:34009 (local address 127.0.0.1:49540)
0927 14:54:02.525451 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0927 14:54:02.525495 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:34009: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260927 14:54:02.526244 13768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:54:02.526028 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:34009 (local address 127.0.0.1:49542)
0927 14:54:02.526158 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0927 14:54:02.526199 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:34009: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80}
W20260927 14:54:02.526400 13765 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:34009: Network error: Client connection negotiation failed: client connection to 127.0.98.60:34009: connect: Connection refused (error 111)
I20260927 14:54:02.557577 13765 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34009 } attempt: 1
I20260927 14:54:02.558625 13768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:54:02.558188 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:34009 (local address 127.0.0.1:49554)
0927 14:54:02.558405 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0927 14:54:02.558499 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:34009: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
I20260927 14:54:02.559491 13768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:54:02.559233 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:34009 (local address 127.0.0.1:49568)
0927 14:54:02.559369 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0927 14:54:02.559424 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:34009: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260927 14:54:02.559636 13765 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:34009: Network error: Client connection negotiation failed: client connection to 127.0.98.60:34009: connect: Connection refused (error 111)
I20260927 14:54:02.571161 13840 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34009 } attempt: 1
I20260927 14:54:02.572049 13843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:54:02.571714 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:34009 (local address 127.0.0.1:49582)
0927 14:54:02.571887 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0927 14:54:02.571958 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:34009: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
I20260927 14:54:02.572765 13843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:54:02.572550 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:34009 (local address 127.0.0.1:49592)
0927 14:54:02.572654 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0927 14:54:02.572706 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:34009: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260927 14:54:02.572907 13840 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:34009: Network error: Client connection negotiation failed: client connection to 127.0.98.60:34009: connect: Connection refused (error 111)
W20260927 14:54:02.627030 13847 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 14:54:02.627265 13847 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 14:54:02.627302 13847 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 14:54:02.631345 13847 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 14:54:02.631407 13847 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 14:54:02.631426 13847 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 14:54:02.631443 13847 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 14:54:02.631457 13847 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 14:54:02.634812 13765 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34009 } attempt: 2
I20260927 14:54:02.635694 13768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:54:02.635370 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:34009 (local address 127.0.0.1:49606)
0927 14:54:02.635537 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0927 14:54:02.635604 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:34009: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20260927 14:54:02.636452 13768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:54:02.636247 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:34009 (local address 127.0.0.1:49612)
0927 14:54:02.636346 (+ 99us) negotiation.cc:107] Waiting for socket to connect
0927 14:54:02.636395 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:34009: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260927 14:54:02.636579 13765 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:34009: Network error: Client connection negotiation failed: client connection to 127.0.98.60:34009: connect: Connection refused (error 111)
I20260927 14:54:02.637296 13847 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34277
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790520684602251-392-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790520684602251-392-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:32875,127.0.98.61:41887,127.0.98.60:34009
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790520684602251-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:34009
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790520684602251-392-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.98.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13847
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790520684602251-392-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:15:17 UTC on ccycloud.quasar-ledxfe.root.comops.site
build id 4676134
I20260927 14:54:02.638284 13847 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 14:54:02.639258 13847 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 14:54:02.641142 13840 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34009 } attempt: 2
I20260927 14:54:02.641948 13843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:54:02.641610 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:34009 (local address 127.0.0.1:49620)
0927 14:54:02.641777 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0927 14:54:02.641851 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:34009: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260927 14:54:02.642705 13843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:54:02.642479 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:34009 (local address 127.0.0.1:49628)
0927 14:54:02.642590 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0927 14:54:02.642641 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:34009: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260927 14:54:02.642868 13840 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:34009: Network error: Client connection negotiation failed: client connection to 127.0.98.60:34009: connect: Connection refused (error 111)
W20260927 14:54:02.648432 13853 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 14:54:02.648452 13856 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 14:54:02.648442 13854 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 14:54:02.648775 13847 server_base.cc:1034] running on GCE node
I20260927 14:54:02.649426 13847 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 14:54:02.650476 13847 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 14:54:02.651664 13847 hybrid_clock.cc:648] HybridClock initialized: now 1790520842651634 us; error 42 us; skew 500 ppm
Sep 27 14:54:02 dist-test-slave-xk1g krb5kdc[13684](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790520842, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 14:54:02.654469 13847 init.cc:377] Logged in from keytab as kudu/127.0.98.60@KRBTEST.COM (short username kudu)
I20260927 14:54:02.655368 13847 webserver.cc:460] Webserver started at http://127.0.98.60:42691/ using document root <none> and password file <none>
I20260927 14:54:02.655918 13847 fs_manager.cc:359] Metadata directory not provided
I20260927 14:54:02.656008 13847 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 14:54:02.656239 13847 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 14:54:02.658296 13847 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790520684602251-392-0/minicluster-data/master-2/data/instance:
uuid: "56a227eb55864f8a9ae516873edb3ed7"
format_stamp: "Formatted at 2026-09-27 14:54:02 on dist-test-slave-xk1g"
server_key: "b6fad20379648a726ae5f797a1e95dfa"
server_key_iv: "a72d087367a063563df49ed08daa6e78"
server_key_version: "encryptionkey@0"
I20260927 14:54:02.658849 13847 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790520684602251-392-0/minicluster-data/master-2/wal/instance:
uuid: "56a227eb55864f8a9ae516873edb3ed7"
format_stamp: "Formatted at 2026-09-27 14:54:02 on dist-test-slave-xk1g"
server_key: "b6fad20379648a726ae5f797a1e95dfa"
server_key_iv: "a72d087367a063563df49ed08daa6e78"
server_key_version: "encryptionkey@0"
I20260927 14:54:02.662570 13847 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.003s
I20260927 14:54:02.664968 13863 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 14:54:02.666185 13847 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260927 14:54:02.666324 13847 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790520684602251-392-0/minicluster-data/master-2/data,/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790520684602251-392-0/minicluster-data/master-2/wal
uuid: "56a227eb55864f8a9ae516873edb3ed7"
format_stamp: "Formatted at 2026-09-27 14:54:02 on dist-test-slave-xk1g"
server_key: "b6fad20379648a726ae5f797a1e95dfa"
server_key_iv: "a72d087367a063563df49ed08daa6e78"
server_key_version: "encryptionkey@0"
I20260927 14:54:02.666438 13847 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790520684602251-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790520684602251-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790520684602251-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 14:54:02.677173 13847 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 14:54:02.680343 13847 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 14:54:02.680532 13847 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 14:54:02.687763 13847 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:34009
I20260927 14:54:02.687778 13915 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:34009 every 8 connection(s)
I20260927 14:54:02.688700 13847 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790520684602251-392-0/minicluster-data/master-2/data/info.pb
I20260927 14:54:02.690364 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6936m5/build/debug/bin/kudu as pid 13847
I20260927 14:54:02.690503 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790520684602251-392-0/minicluster-data/master-2/wal/instance
I20260927 14:54:02.690778 392 external_mini_cluster.cc:1428] Setting key 9cd0f829534ea05840cfddbd8bc377d0
I20260927 14:54:02.691484 13916 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 14:54:02.695315 13916 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32875 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 14:54:02 dist-test-slave-xk1g krb5kdc[13684](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790520842, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
Sep 27 14:54:02 dist-test-slave-xk1g krb5kdc[13684](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790520842, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 14:54:02.704871 13919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:54:02.692141 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49632 (local address 127.0.98.60:34009)
0927 14:54:02.692546 (+ 405us) server_negotiation.cc:206] Beginning negotiation
0927 14:54:02.692560 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 14:54:02.692594 (+ 34us) server_negotiation.cc:407] Connection header received
0927 14:54:02.693300 (+ 706us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:54:02.693319 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:54:02.693632 (+ 313us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:54:02.693963 (+ 331us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:54:02.694752 (+ 789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:54:02.695759 (+ 1007us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:54:02.696613 (+ 854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:54:02.697256 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:54:02.697433 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:54:02.699876 (+ 2443us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:54:02.699882 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:54:02.699890 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 14:54:02.699914 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:54:02.701707 (+ 1793us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:54:02.702209 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:54:02.702215 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:54:02.702220 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:54:02.702366 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:54:02.702689 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:54:02.702691 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:54:02.702693 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:54:02.703238 (+ 545us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:54:02.703469 (+ 231us) server_negotiation.cc:1035] Waiting for connection context
0927 14:54:02.703697 (+ 228us) server_negotiation.cc:299] Negotiation successful
0927 14:54:02.703988 (+ 291us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":276,"thread_start_us":136,"threads_started":1}
I20260927 14:54:02.705621 13920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:54:02.696884 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:32875 (local address 127.0.0.1:50844)
0927 14:54:02.697316 (+ 432us) negotiation.cc:107] Waiting for socket to connect
0927 14:54:02.697376 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0927 14:54:02.697527 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:54:02.697803 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:54:02.697808 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:54:02.698155 (+ 347us) client_negotiation.cc:189] Negotiated authn=SASL
0927 14:54:02.698281 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:54:02.698286 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:54:02.699046 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:54:02.699049 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:54:02.699714 (+ 665us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:54:02.699722 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:54:02.700171 (+ 449us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:54:02.700175 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:54:02.700305 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:54:02.700786 (+ 481us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 14:54:02.700797 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 14:54:02.702163 (+ 1366us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 14:54:02.704052 (+ 1889us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 14:54:02.704060 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 14:54:02.704064 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 14:54:02.704276 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 14:54:02.704552 (+ 276us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 14:54:02.704554 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 14:54:02.704556 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 14:54:02.704636 (+ 80us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 14:54:02.705143 (+ 507us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 14:54:02.705148 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 14:54:02.705272 (+ 124us) client_negotiation.cc:769] Sending connection context
0927 14:54:02.705385 (+ 113us) client_negotiation.cc:240] Negotiation successful
0927 14:54:02.705506 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":248,"thread_start_us":131,"threads_started":1}
I20260927 14:54:02.705749 13769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:54:02.696952 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50844 (local address 127.0.98.62:32875)
0927 14:54:02.697112 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0927 14:54:02.697117 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 14:54:02.697438 (+ 321us) server_negotiation.cc:407] Connection header received
0927 14:54:02.697618 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:54:02.697621 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:54:02.697670 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:54:02.697759 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:54:02.698402 (+ 643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:54:02.698909 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:54:02.699848 (+ 939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:54:02.700062 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:54:02.700172 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:54:02.702292 (+ 2120us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:54:02.702298 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:54:02.702305 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 14:54:02.702349 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:54:02.703928 (+ 1579us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:54:02.704375 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:54:02.704379 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:54:02.704381 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:54:02.704463 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:54:02.704722 (+ 259us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:54:02.704725 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:54:02.704727 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:54:02.704956 (+ 229us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:54:02.705048 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0927 14:54:02.705474 (+ 426us) server_negotiation.cc:299] Negotiation successful
0927 14:54:02.705589 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260927 14:54:02.706763 13916 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41887 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 14:54:02 dist-test-slave-xk1g krb5kdc[13684](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790520842, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
Sep 27 14:54:02 dist-test-slave-xk1g krb5kdc[13684](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790520842, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 14:54:02.715377 13920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:54:02.707297 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:41887 (local address 127.0.0.1:46012)
0927 14:54:02.707421 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0927 14:54:02.707437 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 14:54:02.707627 (+ 190us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:54:02.707990 (+ 363us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:54:02.707992 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:54:02.708121 (+ 129us) client_negotiation.cc:189] Negotiated authn=SASL
0927 14:54:02.708205 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:54:02.708209 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:54:02.708954 (+ 745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:54:02.708959 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:54:02.709538 (+ 579us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:54:02.709546 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:54:02.710018 (+ 472us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:54:02.710021 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:54:02.710112 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:54:02.710588 (+ 476us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 14:54:02.710606 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0927 14:54:02.711826 (+ 1220us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 14:54:02.713728 (+ 1902us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 14:54:02.713733 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 14:54:02.713738 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 14:54:02.713980 (+ 242us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 14:54:02.714290 (+ 310us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 14:54:02.714292 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 14:54:02.714294 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 14:54:02.714408 (+ 114us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 14:54:02.714869 (+ 461us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 14:54:02.714872 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 14:54:02.714969 (+ 97us) client_negotiation.cc:769] Sending connection context
0927 14:54:02.715080 (+ 111us) client_negotiation.cc:240] Negotiation successful
0927 14:54:02.715207 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":55}
I20260927 14:54:02.715454 13846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:54:02.707416 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46012 (local address 127.0.98.61:41887)
0927 14:54:02.707520 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0927 14:54:02.707525 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 14:54:02.707614 (+ 89us) server_negotiation.cc:407] Connection header received
0927 14:54:02.707722 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:54:02.707725 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:54:02.707778 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:54:02.707876 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:54:02.708319 (+ 443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:54:02.708843 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:54:02.709658 (+ 815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:54:02.709908 (+ 250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:54:02.710018 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:54:02.711965 (+ 1947us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:54:02.711969 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:54:02.711976 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 14:54:02.712010 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:54:02.713618 (+ 1608us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:54:02.714087 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:54:02.714091 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:54:02.714093 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:54:02.714182 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:54:02.714529 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:54:02.714532 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:54:02.714534 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:54:02.714775 (+ 241us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:54:02.714849 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0927 14:54:02.715144 (+ 295us) server_negotiation.cc:299] Negotiation successful
0927 14:54:02.715270 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260927 14:54:02.715895 13769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:54:02.707591 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50850 (local address 127.0.98.62:32875)
0927 14:54:02.707750 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0927 14:54:02.707756 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 14:54:02.707770 (+ 14us) server_negotiation.cc:407] Connection header received
0927 14:54:02.707961 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:54:02.707968 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:54:02.708016 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:54:02.708100 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:54:02.708669 (+ 569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:54:02.709201 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:54:02.709938 (+ 737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:54:02.710179 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:54:02.710287 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:54:02.712466 (+ 2179us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:54:02.712471 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:54:02.712479 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 14:54:02.712499 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:54:02.713843 (+ 1344us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:54:02.714349 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:54:02.714355 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:54:02.714358 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:54:02.714432 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:54:02.714707 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:54:02.714709 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:54:02.714710 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:54:02.714909 (+ 199us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:54:02.715151 (+ 242us) server_negotiation.cc:1035] Waiting for connection context
0927 14:54:02.715291 (+ 140us) server_negotiation.cc:299] Negotiation successful
0927 14:54:02.715394 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260927 14:54:02.716254 13916 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34009 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 14:54:02 dist-test-slave-xk1g krb5kdc[13684](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790520842, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
Sep 27 14:54:02 dist-test-slave-xk1g krb5kdc[13684](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790520842, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260927 14:54:02.724905 13920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:54:02.716718 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:34009 (local address 127.0.0.1:49642)
0927 14:54:02.717155 (+ 437us) negotiation.cc:107] Waiting for socket to connect
0927 14:54:02.717174 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 14:54:02.717263 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:54:02.717464 (+ 201us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:54:02.717466 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:54:02.717627 (+ 161us) client_negotiation.cc:189] Negotiated authn=SASL
0927 14:54:02.717683 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:54:02.717687 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:54:02.718714 (+ 1027us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:54:02.718717 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:54:02.719198 (+ 481us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:54:02.719205 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:54:02.719647 (+ 442us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:54:02.719650 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:54:02.719723 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:54:02.720159 (+ 436us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 14:54:02.720169 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0927 14:54:02.721524 (+ 1355us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 14:54:02.723400 (+ 1876us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 14:54:02.723406 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 14:54:02.723409 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 14:54:02.723616 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 14:54:02.723909 (+ 293us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 14:54:02.723911 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 14:54:02.723912 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 14:54:02.723972 (+ 60us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 14:54:02.724380 (+ 408us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 14:54:02.724383 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 14:54:02.724484 (+ 101us) client_negotiation.cc:769] Sending connection context
0927 14:54:02.724608 (+ 124us) client_negotiation.cc:240] Negotiation successful
0927 14:54:02.724750 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":375}
I20260927 14:54:02.725026 13919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:54:02.716808 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49642 (local address 127.0.98.60:34009)
0927 14:54:02.716929 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0927 14:54:02.716933 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 14:54:02.717229 (+ 296us) server_negotiation.cc:407] Connection header received
0927 14:54:02.717330 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:54:02.717335 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:54:02.717367 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:54:02.717454 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:54:02.717787 (+ 333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:54:02.718599 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:54:02.719306 (+ 707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:54:02.719543 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:54:02.719649 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:54:02.721650 (+ 2001us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:54:02.721653 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:54:02.721655 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 14:54:02.721675 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:54:02.723272 (+ 1597us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:54:02.723715 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:54:02.723719 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:54:02.723720 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:54:02.723818 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:54:02.724056 (+ 238us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:54:02.724059 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:54:02.724061 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:54:02.724289 (+ 228us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:54:02.724359 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0927 14:54:02.724661 (+ 302us) server_negotiation.cc:299] Negotiation successful
0927 14:54:02.724829 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":40,"server-negotiator.queue_time_us":50}
I20260927 14:54:02.725983 13846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:54:02.716988 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46028 (local address 127.0.98.61:41887)
0927 14:54:02.717123 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0927 14:54:02.717129 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 14:54:02.717143 (+ 14us) server_negotiation.cc:407] Connection header received
0927 14:54:02.717273 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:54:02.717276 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:54:02.717313 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:54:02.717400 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:54:02.717958 (+ 558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:54:02.718428 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:54:02.719149 (+ 721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:54:02.719378 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:54:02.719482 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:54:02.722329 (+ 2847us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:54:02.722333 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:54:02.722335 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 14:54:02.722358 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:54:02.724152 (+ 1794us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:54:02.724618 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:54:02.724622 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:54:02.724624 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:54:02.724705 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:54:02.725014 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:54:02.725017 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:54:02.725019 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:54:02.725254 (+ 235us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:54:02.725485 (+ 231us) server_negotiation.cc:1035] Waiting for connection context
0927 14:54:02.725627 (+ 142us) server_negotiation.cc:299] Negotiation successful
0927 14:54:02.725733 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260927 14:54:02.728360 13916 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 56a227eb55864f8a9ae516873edb3ed7: Bootstrap starting.
I20260927 14:54:02.729236 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 14:54:02.730952 13916 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 56a227eb55864f8a9ae516873edb3ed7: Neither blocks nor log segments found. Creating new log.
I20260927 14:54:02.731863 13916 log.cc:824] T 00000000000000000000000000000000 P 56a227eb55864f8a9ae516873edb3ed7: Log is configured to *not* fsync() on all Append() calls
I20260927 14:54:02.734025 13840 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34009 } attempt: 3
I20260927 14:54:02.734118 13916 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 56a227eb55864f8a9ae516873edb3ed7: No bootstrap required, opened a new log
I20260927 14:54:02.735702 13765 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34009 } attempt: 3
I20260927 14:54:02.738720 13916 raft_consensus.cc:348] T 00000000000000000000000000000000 P 56a227eb55864f8a9ae516873edb3ed7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1db0dcf3e694425a6dd5f84316ad738" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32875 } } peers { permanent_uuid: "526ee58668d6401e907cc8d97793c839" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41887 } } peers { permanent_uuid: "56a227eb55864f8a9ae516873edb3ed7" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34009 } }
I20260927 14:54:02.739048 13916 raft_consensus.cc:374] T 00000000000000000000000000000000 P 56a227eb55864f8a9ae516873edb3ed7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 14:54:02.739143 13916 raft_consensus.cc:729] T 00000000000000000000000000000000 P 56a227eb55864f8a9ae516873edb3ed7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 56a227eb55864f8a9ae516873edb3ed7, State: Initialized, Role: FOLLOWER
I20260927 14:54:02.739660 13916 consensus_queue.cc:260] T 00000000000000000000000000000000 P 56a227eb55864f8a9ae516873edb3ed7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1db0dcf3e694425a6dd5f84316ad738" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32875 } } peers { permanent_uuid: "526ee58668d6401e907cc8d97793c839" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41887 } } peers { permanent_uuid: "56a227eb55864f8a9ae516873edb3ed7" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34009 } }
I20260927 14:54:02.741146 13916 sys_catalog.cc:564] T 00000000000000000000000000000000 P 56a227eb55864f8a9ae516873edb3ed7 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 14:54:02.742280 13769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:54:02.731186 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50864 (local address 127.0.98.62:32875)
0927 14:54:02.731297 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0927 14:54:02.731303 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 14:54:02.731549 (+ 246us) server_negotiation.cc:407] Connection header received
0927 14:54:02.731732 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:54:02.731737 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:54:02.731788 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:54:02.731848 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:54:02.732783 (+ 935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:54:02.733561 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:54:02.734692 (+ 1131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:54:02.734995 (+ 303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:54:02.735067 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:54:02.736971 (+ 1904us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:54:02.736975 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:54:02.736977 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 14:54:02.736999 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:54:02.738837 (+ 1838us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:54:02.740714 (+ 1877us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:54:02.740723 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:54:02.740725 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:54:02.740815 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:54:02.741189 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:54:02.741196 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:54:02.741203 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:54:02.741449 (+ 246us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:54:02.741555 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0927 14:54:02.741987 (+ 432us) server_negotiation.cc:299] Negotiation successful
0927 14:54:02.742106 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260927 14:54:02.742467 13846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:54:02.731357 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46030 (local address 127.0.98.61:41887)
0927 14:54:02.731449 (+ 92us) server_negotiation.cc:206] Beginning negotiation
0927 14:54:02.731453 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 14:54:02.732101 (+ 648us) server_negotiation.cc:407] Connection header received
0927 14:54:02.732210 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:54:02.732216 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:54:02.732265 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:54:02.732338 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:54:02.733045 (+ 707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:54:02.733738 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:54:02.735824 (+ 2086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:54:02.736158 (+ 334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:54:02.736234 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:54:02.737851 (+ 1617us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:54:02.737858 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:54:02.737863 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 14:54:02.737893 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:54:02.740186 (+ 2293us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:54:02.740810 (+ 624us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:54:02.740813 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:54:02.740815 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:54:02.740896 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:54:02.741234 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:54:02.741238 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:54:02.741239 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:54:02.741429 (+ 190us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:54:02.741556 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0927 14:54:02.742159 (+ 603us) server_negotiation.cc:299] Negotiation successful
0927 14:54:02.742285 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260927 14:54:02.743863 13931 sys_catalog.cc:455] T 00000000000000000000000000000000 P 56a227eb55864f8a9ae516873edb3ed7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1db0dcf3e694425a6dd5f84316ad738" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32875 } } peers { permanent_uuid: "526ee58668d6401e907cc8d97793c839" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41887 } } peers { permanent_uuid: "56a227eb55864f8a9ae516873edb3ed7" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34009 } } }
I20260927 14:54:02.744004 13931 sys_catalog.cc:458] T 00000000000000000000000000000000 P 56a227eb55864f8a9ae516873edb3ed7 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 14:54:02.744405 13919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:54:02.731615 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49658 (local address 127.0.98.60:34009)
0927 14:54:02.731710 (+ 95us) server_negotiation.cc:206] Beginning negotiation
0927 14:54:02.731716 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 14:54:02.733052 (+ 1336us) server_negotiation.cc:407] Connection header received
0927 14:54:02.733136 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:54:02.733139 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:54:02.733177 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:54:02.733239 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:54:02.734045 (+ 806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:54:02.734738 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:54:02.737167 (+ 2429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:54:02.738006 (+ 839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:54:02.738342 (+ 336us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:54:02.740228 (+ 1886us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:54:02.740244 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:54:02.740249 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 14:54:02.740273 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:54:02.741957 (+ 1684us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:54:02.742449 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:54:02.742453 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:54:02.742455 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:54:02.742555 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:54:02.742886 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:54:02.742889 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:54:02.742891 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:54:02.743108 (+ 217us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:54:02.743218 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0927 14:54:02.743841 (+ 623us) server_negotiation.cc:299] Negotiation successful
0927 14:54:02.743955 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
Sep 27 14:54:02 dist-test-slave-xk1g krb5kdc[13684](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790520842, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
W20260927 14:54:02.748533 13942 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 56a227eb55864f8a9ae516873edb3ed7: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 14:54:02.748705 13942 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
Sep 27 14:54:02 dist-test-slave-xk1g krb5kdc[13684](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790520842, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260927 14:54:02.750124 13843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:54:02.734379 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:34009 (local address 127.0.0.1:49670)
0927 14:54:02.734533 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0927 14:54:02.734549 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 14:54:02.734619 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:54:02.738227 (+ 3608us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:54:02.738230 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:54:02.738392 (+ 162us) client_negotiation.cc:189] Negotiated authn=SASL
0927 14:54:02.738475 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:54:02.738481 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:54:02.739544 (+ 1063us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:54:02.739547 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:54:02.740290 (+ 743us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:54:02.740301 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:54:02.743288 (+ 2987us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:54:02.743294 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:54:02.743383 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:54:02.743803 (+ 420us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 14:54:02.743817 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0927 14:54:02.745466 (+ 1649us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 14:54:02.748056 (+ 2590us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 14:54:02.748060 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 14:54:02.748062 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 14:54:02.748307 (+ 245us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 14:54:02.748633 (+ 326us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 14:54:02.748636 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 14:54:02.748637 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 14:54:02.748728 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 14:54:02.749359 (+ 631us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 14:54:02.749363 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 14:54:02.749482 (+ 119us) client_negotiation.cc:769] Sending connection context
0927 14:54:02.749640 (+ 158us) client_negotiation.cc:240] Negotiation successful
0927 14:54:02.749820 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":86}
I20260927 14:54:02.750783 13930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:54:02.734799 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49670 (local address 127.0.98.60:34009)
0927 14:54:02.737982 (+ 3183us) server_negotiation.cc:206] Beginning negotiation
0927 14:54:02.737990 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 14:54:02.738002 (+ 12us) server_negotiation.cc:407] Connection header received
0927 14:54:02.738033 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:54:02.738036 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:54:02.738082 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:54:02.738580 (+ 498us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:54:02.738682 (+ 102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:54:02.739421 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:54:02.742835 (+ 3414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:54:02.743151 (+ 316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:54:02.743572 (+ 421us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:54:02.745659 (+ 2087us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:54:02.745663 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:54:02.745665 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 14:54:02.745686 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:54:02.747940 (+ 2254us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:54:02.748406 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:54:02.748412 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:54:02.748414 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:54:02.748511 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:54:02.748861 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:54:02.748865 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:54:02.748868 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:54:02.749180 (+ 312us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:54:02.750406 (+ 1226us) server_negotiation.cc:1035] Waiting for connection context
0927 14:54:02.750499 (+ 93us) server_negotiation.cc:299] Negotiation successful
0927 14:54:02.750626 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3117,"thread_start_us":105,"threads_started":1}
I20260927 14:54:02.752823 13768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:54:02.736045 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:34009 (local address 127.0.0.1:49678)
0927 14:54:02.736289 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0927 14:54:02.736304 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 14:54:02.736397 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:54:02.744837 (+ 8440us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:54:02.744840 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:54:02.745013 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0927 14:54:02.745179 (+ 166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:54:02.745188 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:54:02.746121 (+ 933us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:54:02.746124 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:54:02.746860 (+ 736us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:54:02.746871 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:54:02.747416 (+ 545us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:54:02.747420 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:54:02.747494 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:54:02.747944 (+ 450us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 14:54:02.747958 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0927 14:54:02.749373 (+ 1415us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 14:54:02.751296 (+ 1923us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 14:54:02.751300 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 14:54:02.751302 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 14:54:02.751534 (+ 232us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 14:54:02.751866 (+ 332us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 14:54:02.751869 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 14:54:02.751871 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 14:54:02.751956 (+ 85us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 14:54:02.752354 (+ 398us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 14:54:02.752357 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 14:54:02.752446 (+ 89us) client_negotiation.cc:769] Sending connection context
0927 14:54:02.752529 (+ 83us) client_negotiation.cc:240] Negotiation successful
0927 14:54:02.752668 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":166}
I20260927 14:54:02.749320 13847 master_runner.cc:186] Error getting master registration for 127.0.98.62:32875: OK, instance_id { permanent_uuid: "a1db0dcf3e694425a6dd5f84316ad738" instance_seqno: 1790520842268111 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260927 14:54:02.753348 13847 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260927 14:54:02.753743 13919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:54:02.736532 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49678 (local address 127.0.98.60:34009)
0927 14:54:02.744560 (+ 8028us) server_negotiation.cc:206] Beginning negotiation
0927 14:54:02.744565 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 14:54:02.744579 (+ 14us) server_negotiation.cc:407] Connection header received
0927 14:54:02.744618 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:54:02.744622 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:54:02.744661 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:54:02.744786 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:54:02.745310 (+ 524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:54:02.745978 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:54:02.746999 (+ 1021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:54:02.747293 (+ 294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:54:02.747685 (+ 392us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:54:02.749527 (+ 1842us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:54:02.749533 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:54:02.749537 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 14:54:02.749568 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:54:02.751167 (+ 1599us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:54:02.751667 (+ 500us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:54:02.751670 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:54:02.751671 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:54:02.751754 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:54:02.752065 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:54:02.752068 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:54:02.752069 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:54:02.752246 (+ 177us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:54:02.753373 (+ 1127us) server_negotiation.cc:1035] Waiting for connection context
0927 14:54:02.753491 (+ 118us) server_negotiation.cc:299] Negotiation successful
0927 14:54:02.753591 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7974,"thread_start_us":377,"threads_started":1}
I20260927 14:54:02.753873 13840 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 526ee58668d6401e907cc8d97793c839: Bootstrap starting.
I20260927 14:54:02.756544 13840 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 526ee58668d6401e907cc8d97793c839: Neither blocks nor log segments found. Creating new log.
I20260927 14:54:02.756891 13765 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a1db0dcf3e694425a6dd5f84316ad738: Bootstrap starting.
I20260927 14:54:02.757444 13840 log.cc:824] T 00000000000000000000000000000000 P 526ee58668d6401e907cc8d97793c839: Log is configured to *not* fsync() on all Append() calls
I20260927 14:54:02.759426 13765 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a1db0dcf3e694425a6dd5f84316ad738: Neither blocks nor log segments found. Creating new log.
I20260927 14:54:02.759702 13840 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 526ee58668d6401e907cc8d97793c839: No bootstrap required, opened a new log
I20260927 14:54:02.760268 13765 log.cc:824] T 00000000000000000000000000000000 P a1db0dcf3e694425a6dd5f84316ad738: Log is configured to *not* fsync() on all Append() calls
I20260927 14:54:02.762306 13765 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a1db0dcf3e694425a6dd5f84316ad738: No bootstrap required, opened a new log
I20260927 14:54:02.763713 13840 raft_consensus.cc:348] T 00000000000000000000000000000000 P 526ee58668d6401e907cc8d97793c839 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1db0dcf3e694425a6dd5f84316ad738" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32875 } } peers { permanent_uuid: "526ee58668d6401e907cc8d97793c839" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41887 } } peers { permanent_uuid: "56a227eb55864f8a9ae516873edb3ed7" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34009 } }
I20260927 14:54:02.763943 13840 raft_consensus.cc:374] T 00000000000000000000000000000000 P 526ee58668d6401e907cc8d97793c839 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 14:54:02.763988 13840 raft_consensus.cc:729] T 00000000000000000000000000000000 P 526ee58668d6401e907cc8d97793c839 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 526ee58668d6401e907cc8d97793c839, State: Initialized, Role: FOLLOWER
I20260927 14:54:02.764580 13840 consensus_queue.cc:260] T 00000000000000000000000000000000 P 526ee58668d6401e907cc8d97793c839 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1db0dcf3e694425a6dd5f84316ad738" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32875 } } peers { permanent_uuid: "526ee58668d6401e907cc8d97793c839" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41887 } } peers { permanent_uuid: "56a227eb55864f8a9ae516873edb3ed7" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34009 } }
I20260927 14:54:02.765920 13840 sys_catalog.cc:564] T 00000000000000000000000000000000 P 526ee58668d6401e907cc8d97793c839 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 14:54:02.766162 13765 raft_consensus.cc:348] T 00000000000000000000000000000000 P a1db0dcf3e694425a6dd5f84316ad738 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1db0dcf3e694425a6dd5f84316ad738" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32875 } } peers { permanent_uuid: "526ee58668d6401e907cc8d97793c839" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41887 } } peers { permanent_uuid: "56a227eb55864f8a9ae516873edb3ed7" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34009 } }
I20260927 14:54:02.766379 13765 raft_consensus.cc:374] T 00000000000000000000000000000000 P a1db0dcf3e694425a6dd5f84316ad738 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 14:54:02.766435 13765 raft_consensus.cc:729] T 00000000000000000000000000000000 P a1db0dcf3e694425a6dd5f84316ad738 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a1db0dcf3e694425a6dd5f84316ad738, State: Initialized, Role: FOLLOWER
I20260927 14:54:02.766889 13765 consensus_queue.cc:260] T 00000000000000000000000000000000 P a1db0dcf3e694425a6dd5f84316ad738 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1db0dcf3e694425a6dd5f84316ad738" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32875 } } peers { permanent_uuid: "526ee58668d6401e907cc8d97793c839" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41887 } } peers { permanent_uuid: "56a227eb55864f8a9ae516873edb3ed7" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34009 } }
I20260927 14:54:02.767357 13952 sys_catalog.cc:455] T 00000000000000000000000000000000 P a1db0dcf3e694425a6dd5f84316ad738 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1db0dcf3e694425a6dd5f84316ad738" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32875 } } peers { permanent_uuid: "526ee58668d6401e907cc8d97793c839" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41887 } } peers { permanent_uuid: "56a227eb55864f8a9ae516873edb3ed7" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34009 } } }
I20260927 14:54:02.767530 13952 sys_catalog.cc:458] T 00000000000000000000000000000000 P a1db0dcf3e694425a6dd5f84316ad738 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 14:54:02.768231 13765 sys_catalog.cc:564] T 00000000000000000000000000000000 P a1db0dcf3e694425a6dd5f84316ad738 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 14:54:02.769701 13772 master_runner.cc:186] Error getting master registration for 127.0.98.62:32875: OK, instance_id { permanent_uuid: "a1db0dcf3e694425a6dd5f84316ad738" instance_seqno: 1790520842268111 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
W20260927 14:54:02.772647 13958 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 526ee58668d6401e907cc8d97793c839: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
I20260927 14:54:02.772768 13772 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260927 14:54:02.772778 13958 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 14:54:02.773223 13946 sys_catalog.cc:455] T 00000000000000000000000000000000 P 526ee58668d6401e907cc8d97793c839 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1db0dcf3e694425a6dd5f84316ad738" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32875 } } peers { permanent_uuid: "526ee58668d6401e907cc8d97793c839" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41887 } } peers { permanent_uuid: "56a227eb55864f8a9ae516873edb3ed7" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34009 } } }
I20260927 14:54:02.773345 13946 sys_catalog.cc:458] T 00000000000000000000000000000000 P 526ee58668d6401e907cc8d97793c839 [sys.catalog]: This master's current role is: FOLLOWER
W20260927 14:54:02.776083 13970 catalog_manager.cc:1558] T 00000000000000000000000000000000 P a1db0dcf3e694425a6dd5f84316ad738: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 14:54:02.776197 13970 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 14:54:02.778815 13697 master_runner.cc:263] No leader master found from master a1db0dcf3e694425a6dd5f84316ad738
I20260927 14:54:02.778874 13697 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260927 14:54:02.931970 13869 tablet.cc:2369] T 00000000000000000000000000000000 P 56a227eb55864f8a9ae516873edb3ed7: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 14:54:02.975049 13793 tablet.cc:2369] T 00000000000000000000000000000000 P 526ee58668d6401e907cc8d97793c839: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 14:54:03.014792 13718 tablet.cc:2369] T 00000000000000000000000000000000 P a1db0dcf3e694425a6dd5f84316ad738: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 14:54:03.137701 13952 raft_consensus.cc:482] T 00000000000000000000000000000000 P a1db0dcf3e694425a6dd5f84316ad738 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 14:54:03.138140 13952 raft_consensus.cc:504] T 00000000000000000000000000000000 P a1db0dcf3e694425a6dd5f84316ad738 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1db0dcf3e694425a6dd5f84316ad738" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32875 } } peers { permanent_uuid: "526ee58668d6401e907cc8d97793c839" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41887 } } peers { permanent_uuid: "56a227eb55864f8a9ae516873edb3ed7" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34009 } }
I20260927 14:54:03.139161 13952 leader_election.cc:288] T 00000000000000000000000000000000 P a1db0dcf3e694425a6dd5f84316ad738 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 526ee58668d6401e907cc8d97793c839 (127.0.98.61:41887), 56a227eb55864f8a9ae516873edb3ed7 (127.0.98.60:34009)
I20260927 14:54:03.139765 13814 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "a1db0dcf3e694425a6dd5f84316ad738" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "526ee58668d6401e907cc8d97793c839" is_pre_election: true
I20260927 14:54:03.139849 13890 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "a1db0dcf3e694425a6dd5f84316ad738" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "56a227eb55864f8a9ae516873edb3ed7" is_pre_election: true
I20260927 14:54:03.140085 13814 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 526ee58668d6401e907cc8d97793c839 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a1db0dcf3e694425a6dd5f84316ad738 in term 0.
I20260927 14:54:03.140218 13890 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 56a227eb55864f8a9ae516873edb3ed7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a1db0dcf3e694425a6dd5f84316ad738 in term 0.
I20260927 14:54:03.140527 13713 leader_election.cc:302] T 00000000000000000000000000000000 P a1db0dcf3e694425a6dd5f84316ad738 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 526ee58668d6401e907cc8d97793c839, a1db0dcf3e694425a6dd5f84316ad738; no voters:
I20260927 14:54:03.140800 13952 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a1db0dcf3e694425a6dd5f84316ad738 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 14:54:03.140889 13952 raft_consensus.cc:482] T 00000000000000000000000000000000 P a1db0dcf3e694425a6dd5f84316ad738 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 14:54:03.140935 13952 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a1db0dcf3e694425a6dd5f84316ad738 [term 0 FOLLOWER]: Advancing to term 1
I20260927 14:54:03.142047 13952 raft_consensus.cc:504] T 00000000000000000000000000000000 P a1db0dcf3e694425a6dd5f84316ad738 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1db0dcf3e694425a6dd5f84316ad738" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32875 } } peers { permanent_uuid: "526ee58668d6401e907cc8d97793c839" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41887 } } peers { permanent_uuid: "56a227eb55864f8a9ae516873edb3ed7" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34009 } }
I20260927 14:54:03.142385 13952 leader_election.cc:288] T 00000000000000000000000000000000 P a1db0dcf3e694425a6dd5f84316ad738 [CANDIDATE]: Term 1 election: Requested vote from peers 526ee58668d6401e907cc8d97793c839 (127.0.98.61:41887), 56a227eb55864f8a9ae516873edb3ed7 (127.0.98.60:34009)
I20260927 14:54:03.142704 13814 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "a1db0dcf3e694425a6dd5f84316ad738" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "526ee58668d6401e907cc8d97793c839"
I20260927 14:54:03.142854 13814 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 526ee58668d6401e907cc8d97793c839 [term 0 FOLLOWER]: Advancing to term 1
I20260927 14:54:03.142938 13890 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "a1db0dcf3e694425a6dd5f84316ad738" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "56a227eb55864f8a9ae516873edb3ed7"
I20260927 14:54:03.143117 13890 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 56a227eb55864f8a9ae516873edb3ed7 [term 0 FOLLOWER]: Advancing to term 1
I20260927 14:54:03.143958 13814 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 526ee58668d6401e907cc8d97793c839 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a1db0dcf3e694425a6dd5f84316ad738 in term 1.
I20260927 14:54:03.144207 13890 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 56a227eb55864f8a9ae516873edb3ed7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a1db0dcf3e694425a6dd5f84316ad738 in term 1.
I20260927 14:54:03.144311 13713 leader_election.cc:302] T 00000000000000000000000000000000 P a1db0dcf3e694425a6dd5f84316ad738 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 526ee58668d6401e907cc8d97793c839, a1db0dcf3e694425a6dd5f84316ad738; no voters:
I20260927 14:54:03.144480 13952 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a1db0dcf3e694425a6dd5f84316ad738 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 14:54:03.144690 13952 raft_consensus.cc:686] T 00000000000000000000000000000000 P a1db0dcf3e694425a6dd5f84316ad738 [term 1 LEADER]: Becoming Leader. State: Replica: a1db0dcf3e694425a6dd5f84316ad738, State: Running, Role: LEADER
I20260927 14:54:03.145016 13952 consensus_queue.cc:237] T 00000000000000000000000000000000 P a1db0dcf3e694425a6dd5f84316ad738 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1db0dcf3e694425a6dd5f84316ad738" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32875 } } peers { permanent_uuid: "526ee58668d6401e907cc8d97793c839" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41887 } } peers { permanent_uuid: "56a227eb55864f8a9ae516873edb3ed7" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34009 } }
I20260927 14:54:03.146981 13972 sys_catalog.cc:455] T 00000000000000000000000000000000 P a1db0dcf3e694425a6dd5f84316ad738 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a1db0dcf3e694425a6dd5f84316ad738. Latest consensus state: current_term: 1 leader_uuid: "a1db0dcf3e694425a6dd5f84316ad738" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1db0dcf3e694425a6dd5f84316ad738" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32875 } } peers { permanent_uuid: "526ee58668d6401e907cc8d97793c839" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41887 } } peers { permanent_uuid: "56a227eb55864f8a9ae516873edb3ed7" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34009 } } }
I20260927 14:54:03.147099 13972 sys_catalog.cc:458] T 00000000000000000000000000000000 P a1db0dcf3e694425a6dd5f84316ad738 [sys.catalog]: This master's current role is: LEADER
I20260927 14:54:03.147477 13974 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 14:54:03.148593 13974 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 14:54:03.152936 13890 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 56a227eb55864f8a9ae516873edb3ed7 [term 1 FOLLOWER]: Refusing update from remote peer a1db0dcf3e694425a6dd5f84316ad738: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 14:54:03.152978 13814 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 526ee58668d6401e907cc8d97793c839 [term 1 FOLLOWER]: Refusing update from remote peer a1db0dcf3e694425a6dd5f84316ad738: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 14:54:03.153613 13972 consensus_queue.cc:1035] T 00000000000000000000000000000000 P a1db0dcf3e694425a6dd5f84316ad738 [LEADER]: Connected to new peer: Peer: permanent_uuid: "526ee58668d6401e907cc8d97793c839" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41887 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 14:54:03.153801 13952 consensus_queue.cc:1035] T 00000000000000000000000000000000 P a1db0dcf3e694425a6dd5f84316ad738 [LEADER]: Connected to new peer: Peer: permanent_uuid: "56a227eb55864f8a9ae516873edb3ed7" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34009 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 14:54:03.157227 13946 sys_catalog.cc:455] T 00000000000000000000000000000000 P 526ee58668d6401e907cc8d97793c839 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a1db0dcf3e694425a6dd5f84316ad738. Latest consensus state: current_term: 1 leader_uuid: "a1db0dcf3e694425a6dd5f84316ad738" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1db0dcf3e694425a6dd5f84316ad738" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32875 } } peers { permanent_uuid: "526ee58668d6401e907cc8d97793c839" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41887 } } peers { permanent_uuid: "56a227eb55864f8a9ae516873edb3ed7" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34009 } } }
I20260927 14:54:03.157343 13946 sys_catalog.cc:458] T 00000000000000000000000000000000 P 526ee58668d6401e907cc8d97793c839 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 14:54:03.157338 13931 sys_catalog.cc:455] T 00000000000000000000000000000000 P 56a227eb55864f8a9ae516873edb3ed7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a1db0dcf3e694425a6dd5f84316ad738. Latest consensus state: current_term: 1 leader_uuid: "a1db0dcf3e694425a6dd5f84316ad738" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1db0dcf3e694425a6dd5f84316ad738" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32875 } } peers { permanent_uuid: "526ee58668d6401e907cc8d97793c839" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41887 } } peers { permanent_uuid: "56a227eb55864f8a9ae516873edb3ed7" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34009 } } }
I20260927 14:54:03.157444 13931 sys_catalog.cc:458] T 00000000000000000000000000000000 P 56a227eb55864f8a9ae516873edb3ed7 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 14:54:03.159070 13952 sys_catalog.cc:455] T 00000000000000000000000000000000 P a1db0dcf3e694425a6dd5f84316ad738 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "a1db0dcf3e694425a6dd5f84316ad738" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1db0dcf3e694425a6dd5f84316ad738" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32875 } } peers { permanent_uuid: "526ee58668d6401e907cc8d97793c839" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41887 } } peers { permanent_uuid: "56a227eb55864f8a9ae516873edb3ed7" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34009 } } }
I20260927 14:54:03.159185 13952 sys_catalog.cc:458] T 00000000000000000000000000000000 P a1db0dcf3e694425a6dd5f84316ad738 [sys.catalog]: This master's current role is: LEADER
I20260927 14:54:03.159274 13975 mvcc.cc:205] Tried to move back new op lower bound from 7333973373548191744 to 7333973373524881408. Current Snapshot: MvccSnapshot[applied={T|T < 7333973373548191744}]
I20260927 14:54:03.159313 13946 sys_catalog.cc:455] T 00000000000000000000000000000000 P 526ee58668d6401e907cc8d97793c839 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "a1db0dcf3e694425a6dd5f84316ad738" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1db0dcf3e694425a6dd5f84316ad738" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32875 } } peers { permanent_uuid: "526ee58668d6401e907cc8d97793c839" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41887 } } peers { permanent_uuid: "56a227eb55864f8a9ae516873edb3ed7" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34009 } } }
I20260927 14:54:03.159437 13946 sys_catalog.cc:458] T 00000000000000000000000000000000 P 526ee58668d6401e907cc8d97793c839 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 14:54:03.159942 13972 sys_catalog.cc:455] T 00000000000000000000000000000000 P a1db0dcf3e694425a6dd5f84316ad738 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "a1db0dcf3e694425a6dd5f84316ad738" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1db0dcf3e694425a6dd5f84316ad738" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32875 } } peers { permanent_uuid: "526ee58668d6401e907cc8d97793c839" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41887 } } peers { permanent_uuid: "56a227eb55864f8a9ae516873edb3ed7" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34009 } } }
I20260927 14:54:03.160066 13972 sys_catalog.cc:458] T 00000000000000000000000000000000 P a1db0dcf3e694425a6dd5f84316ad738 [sys.catalog]: This master's current role is: LEADER
I20260927 14:54:03.160938 13931 sys_catalog.cc:455] T 00000000000000000000000000000000 P 56a227eb55864f8a9ae516873edb3ed7 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "a1db0dcf3e694425a6dd5f84316ad738" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1db0dcf3e694425a6dd5f84316ad738" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32875 } } peers { permanent_uuid: "526ee58668d6401e907cc8d97793c839" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41887 } } peers { permanent_uuid: "56a227eb55864f8a9ae516873edb3ed7" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34009 } } }
I20260927 14:54:03.161078 13931 sys_catalog.cc:458] T 00000000000000000000000000000000 P 56a227eb55864f8a9ae516873edb3ed7 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 14:54:03.161240 13974 catalog_manager.cc:1347] Generated new cluster ID: aafb1a00edc146f7941cac947e3f67a6
I20260927 14:54:03.161305 13974 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 14:54:03.190114 13974 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 14:54:03.191344 13974 catalog_manager.cc:1504] Loading token signing keys...
I20260927 14:54:03.198829 13974 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a1db0dcf3e694425a6dd5f84316ad738: Generated new TSK 0
I20260927 14:54:03.199680 13974 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 14:54:03.750030 13942 catalog_manager.cc:1259] Loaded cluster ID: aafb1a00edc146f7941cac947e3f67a6
I20260927 14:54:03.750104 13942 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 56a227eb55864f8a9ae516873edb3ed7: loading cluster ID for follower catalog manager: success
I20260927 14:54:03.751762 13942 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 56a227eb55864f8a9ae516873edb3ed7: acquiring CA information for follower catalog manager: success
I20260927 14:54:03.752913 13942 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 56a227eb55864f8a9ae516873edb3ed7: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260927 14:54:03.774036 13958 catalog_manager.cc:1259] Loaded cluster ID: aafb1a00edc146f7941cac947e3f67a6
I20260927 14:54:03.774112 13958 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 526ee58668d6401e907cc8d97793c839: loading cluster ID for follower catalog manager: success
I20260927 14:54:03.775713 13958 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 526ee58668d6401e907cc8d97793c839: acquiring CA information for follower catalog manager: success
I20260927 14:54:03.776937 13958 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 526ee58668d6401e907cc8d97793c839: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260927 14:54:04.253091 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6936m5/build/debug/bin/kudu with pid 13697
I20260927 14:54:04.259305 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6936m5/build/debug/bin/kudu with pid 13772
I20260927 14:54:04.265151 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6936m5/build/debug/bin/kudu with pid 13847
2026-09-27T14:54:04Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (3344 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790520684602251-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 14:54:04 dist-test-slave-xk1g krb5kdc[13989](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 14:54:04 dist-test-slave-xk1g krb5kdc[13989](info): set up 2 sockets
Sep 27 14:54:04 dist-test-slave-xk1g krb5kdc[13989](info): commencing operation
krb5kdc: starting...
W20260927 14:54:05.297294 392 mini_kdc.cc:121] Time spent starting KDC: real 1.011s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790520684602251-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790520684602251-392-0/krb5kdc/test-user.keytab.
Sep 27 14:54:05 dist-test-slave-xk1g krb5kdc[13989](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790520845, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T14:54:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T14:54:05Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790520684602251-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790520684602251-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 14:54:05.414373 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6936m5/build/debug/bin/kudu
/tmp/dist-test-task6936m5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790520684602251-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790520684602251-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790520684602251-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790520684602251-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41137
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790520684602251-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34367
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:41137
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790520684602251-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790520684602251-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790520684602251-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 14:54:05.522437 14002 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 14:54:05.522739 14002 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 14:54:05.522785 14002 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 14:54:05.527067 14002 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 14:54:05.527143 14002 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 14:54:05.527165 14002 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 14:54:05.527185 14002 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 14:54:05.527249 14002 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 14:54:05.532618 14002 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34367
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790520684602251-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790520684602251-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:41137
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790520684602251-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41137
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790520684602251-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14002
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790520684602251-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:15:17 UTC on ccycloud.quasar-ledxfe.root.comops.site
build id 4676134
I20260927 14:54:05.533834 14002 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 14:54:05.534900 14002 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 14:54:05.543931 14008 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 14:54:05.543931 14007 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 14:54:05.544128 14002 server_base.cc:1034] running on GCE node
W20260927 14:54:05.543988 14010 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 14:54:05.544802 14002 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 14:54:05.545922 14002 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 14:54:05.547107 14002 hybrid_clock.cc:648] HybridClock initialized: now 1790520845547094 us; error 26 us; skew 500 ppm
Sep 27 14:54:05 dist-test-slave-xk1g krb5kdc[13989](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790520845, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 14:54:05.549777 14002 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 14:54:05.550570 14002 webserver.cc:460] Webserver started at http://127.0.98.62:43249/ using document root <none> and password file <none>
I20260927 14:54:05.551081 14002 fs_manager.cc:359] Metadata directory not provided
I20260927 14:54:05.551137 14002 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 14:54:05.551297 14002 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 14:54:05.553323 14002 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790520684602251-392-0/minicluster-data/master-0/data/instance:
uuid: "c0d61c91713e4b22a411beac60a7f265"
format_stamp: "Formatted at 2026-09-27 14:54:05 on dist-test-slave-xk1g"
server_key: "b79c1f601b03c93c5dc354b68ac1bfa1"
server_key_iv: "3fc0a345fe9ee2c48013e55a7d91afbc"
server_key_version: "encryptionkey@0"
I20260927 14:54:05.553836 14002 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790520684602251-392-0/minicluster-data/master-0/wal/instance:
uuid: "c0d61c91713e4b22a411beac60a7f265"
format_stamp: "Formatted at 2026-09-27 14:54:05 on dist-test-slave-xk1g"
server_key: "b79c1f601b03c93c5dc354b68ac1bfa1"
server_key_iv: "3fc0a345fe9ee2c48013e55a7d91afbc"
server_key_version: "encryptionkey@0"
I20260927 14:54:05.557399 14002 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 14:54:05.559800 14017 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 14:54:05.560992 14002 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 14:54:05.561161 14002 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790520684602251-392-0/minicluster-data/master-0/data,/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790520684602251-392-0/minicluster-data/master-0/wal
uuid: "c0d61c91713e4b22a411beac60a7f265"
format_stamp: "Formatted at 2026-09-27 14:54:05 on dist-test-slave-xk1g"
server_key: "b79c1f601b03c93c5dc354b68ac1bfa1"
server_key_iv: "3fc0a345fe9ee2c48013e55a7d91afbc"
server_key_version: "encryptionkey@0"
I20260927 14:54:05.561272 14002 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790520684602251-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790520684602251-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790520684602251-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 14:54:05.576383 14002 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 14:54:05.579262 14002 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 14:54:05.579408 14002 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 14:54:05.585891 14002 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:41137
I20260927 14:54:05.585913 14069 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:41137 every 8 connection(s)
I20260927 14:54:05.586779 14002 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790520684602251-392-0/minicluster-data/master-0/data/info.pb
I20260927 14:54:05.589483 14070 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 14:54:05.590235 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6936m5/build/debug/bin/kudu as pid 14002
I20260927 14:54:05.590327 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790520684602251-392-0/minicluster-data/master-0/wal/instance
I20260927 14:54:05.590613 392 external_mini_cluster.cc:1428] Setting key 9db6354a3129e31677e97e9ca0eb958b
I20260927 14:54:05.595276 14070 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c0d61c91713e4b22a411beac60a7f265: Bootstrap starting.
I20260927 14:54:05.597525 14070 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c0d61c91713e4b22a411beac60a7f265: Neither blocks nor log segments found. Creating new log.
I20260927 14:54:05.598268 14070 log.cc:824] T 00000000000000000000000000000000 P c0d61c91713e4b22a411beac60a7f265: Log is configured to *not* fsync() on all Append() calls
Sep 27 14:54:05 dist-test-slave-xk1g krb5kdc[13989](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790520845, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 14:54:05.600370 14070 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c0d61c91713e4b22a411beac60a7f265: No bootstrap required, opened a new log
I20260927 14:54:05.604707 14073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:54:05.591906 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46828 (local address 127.0.98.62:41137)
0927 14:54:05.592322 (+ 416us) server_negotiation.cc:206] Beginning negotiation
0927 14:54:05.592335 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 14:54:05.592368 (+ 33us) server_negotiation.cc:407] Connection header received
0927 14:54:05.592992 (+ 624us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:54:05.593008 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:54:05.593388 (+ 380us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:54:05.593728 (+ 340us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:54:05.594506 (+ 778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:54:05.595500 (+ 994us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:54:05.596331 (+ 831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:54:05.596798 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:54:05.596961 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:54:05.599458 (+ 2497us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:54:05.599464 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:54:05.599480 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 14:54:05.599508 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:54:05.601385 (+ 1877us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:54:05.601795 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:54:05.601801 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:54:05.601807 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:54:05.601941 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:54:05.602185 (+ 244us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:54:05.602190 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:54:05.602193 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:54:05.602753 (+ 560us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:54:05.602994 (+ 241us) server_negotiation.cc:1035] Waiting for connection context
0927 14:54:05.603430 (+ 436us) server_negotiation.cc:299] Negotiation successful
0927 14:54:05.603730 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":285,"thread_start_us":130,"threads_started":1}
I20260927 14:54:05.604837 14070 raft_consensus.cc:348] T 00000000000000000000000000000000 P c0d61c91713e4b22a411beac60a7f265 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0d61c91713e4b22a411beac60a7f265" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41137 } }
I20260927 14:54:05.605099 14070 raft_consensus.cc:374] T 00000000000000000000000000000000 P c0d61c91713e4b22a411beac60a7f265 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 14:54:05.605170 14070 raft_consensus.cc:729] T 00000000000000000000000000000000 P c0d61c91713e4b22a411beac60a7f265 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c0d61c91713e4b22a411beac60a7f265, State: Initialized, Role: FOLLOWER
I20260927 14:54:05.605650 14070 consensus_queue.cc:260] T 00000000000000000000000000000000 P c0d61c91713e4b22a411beac60a7f265 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0d61c91713e4b22a411beac60a7f265" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41137 } }
I20260927 14:54:05.605811 14070 raft_consensus.cc:388] T 00000000000000000000000000000000 P c0d61c91713e4b22a411beac60a7f265 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 14:54:05.605909 14070 raft_consensus.cc:482] T 00000000000000000000000000000000 P c0d61c91713e4b22a411beac60a7f265 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 14:54:05.606004 14070 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c0d61c91713e4b22a411beac60a7f265 [term 0 FOLLOWER]: Advancing to term 1
I20260927 14:54:05.607009 14070 raft_consensus.cc:504] T 00000000000000000000000000000000 P c0d61c91713e4b22a411beac60a7f265 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0d61c91713e4b22a411beac60a7f265" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41137 } }
I20260927 14:54:05.607383 14070 leader_election.cc:302] T 00000000000000000000000000000000 P c0d61c91713e4b22a411beac60a7f265 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c0d61c91713e4b22a411beac60a7f265; no voters:
I20260927 14:54:05.607653 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 14:54:05.607707 14070 leader_election.cc:288] T 00000000000000000000000000000000 P c0d61c91713e4b22a411beac60a7f265 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 14:54:05.607838 14075 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c0d61c91713e4b22a411beac60a7f265 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 14:54:05.608093 14075 raft_consensus.cc:686] T 00000000000000000000000000000000 P c0d61c91713e4b22a411beac60a7f265 [term 1 LEADER]: Becoming Leader. State: Replica: c0d61c91713e4b22a411beac60a7f265, State: Running, Role: LEADER
I20260927 14:54:05.608433 14075 consensus_queue.cc:237] T 00000000000000000000000000000000 P c0d61c91713e4b22a411beac60a7f265 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0d61c91713e4b22a411beac60a7f265" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41137 } }
I20260927 14:54:05.608978 14070 sys_catalog.cc:564] T 00000000000000000000000000000000 P c0d61c91713e4b22a411beac60a7f265 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 14:54:05.610168 14076 sys_catalog.cc:455] T 00000000000000000000000000000000 P c0d61c91713e4b22a411beac60a7f265 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c0d61c91713e4b22a411beac60a7f265. Latest consensus state: current_term: 1 leader_uuid: "c0d61c91713e4b22a411beac60a7f265" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0d61c91713e4b22a411beac60a7f265" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41137 } } }
I20260927 14:54:05.610188 14077 sys_catalog.cc:455] T 00000000000000000000000000000000 P c0d61c91713e4b22a411beac60a7f265 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c0d61c91713e4b22a411beac60a7f265" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0d61c91713e4b22a411beac60a7f265" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41137 } } }
I20260927 14:54:05.610347 14076 sys_catalog.cc:458] T 00000000000000000000000000000000 P c0d61c91713e4b22a411beac60a7f265 [sys.catalog]: This master's current role is: LEADER
I20260927 14:54:05.610410 14077 sys_catalog.cc:458] T 00000000000000000000000000000000 P c0d61c91713e4b22a411beac60a7f265 [sys.catalog]: This master's current role is: LEADER
I20260927 14:54:05.610785 14085 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 14:54:05.613533 14085 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 14:54:05.618397 14085 catalog_manager.cc:1347] Generated new cluster ID: aa2f486faca74d6db25720876a5adca5
I20260927 14:54:05.618466 14085 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 14:54:05.643029 14085 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 14:54:05.643975 14085 catalog_manager.cc:1504] Loading token signing keys...
I20260927 14:54:05.660897 14085 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c0d61c91713e4b22a411beac60a7f265: Generated new TSK 0
I20260927 14:54:05.661626 14085 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 14:54:05.714497 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6936m5/build/debug/bin/kudu with pid 14002
2026-09-27T14:54:05Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (1447 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790520684602251-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 14:54:05 dist-test-slave-xk1g krb5kdc[14097](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 14:54:05 dist-test-slave-xk1g krb5kdc[14097](info): set up 2 sockets
Sep 27 14:54:05 dist-test-slave-xk1g krb5kdc[14097](info): commencing operation
krb5kdc: starting...
W20260927 14:54:06.745360 392 mini_kdc.cc:121] Time spent starting KDC: real 1.012s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790520684602251-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790520684602251-392-0/krb5kdc/test-user.keytab.
Sep 27 14:54:06 dist-test-slave-xk1g krb5kdc[14097](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790520846, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T14:54:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T14:54:06Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790520684602251-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790520684602251-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 14:54:06.865739 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6936m5/build/debug/bin/kudu
/tmp/dist-test-task6936m5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790520684602251-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790520684602251-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790520684602251-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790520684602251-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40583
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790520684602251-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45035
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:40583
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790520684602251-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790520684602251-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790520684602251-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 14:54:06.975320 14110 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 14:54:06.975605 14110 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 14:54:06.975647 14110 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 14:54:06.979763 14110 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 14:54:06.979822 14110 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 14:54:06.979841 14110 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 14:54:06.979857 14110 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 14:54:06.979872 14110 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 14:54:06.985244 14110 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45035
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790520684602251-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790520684602251-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:40583
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790520684602251-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40583
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790520684602251-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14110
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790520684602251-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:15:17 UTC on ccycloud.quasar-ledxfe.root.comops.site
build id 4676134
I20260927 14:54:06.986342 14110 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 14:54:06.987373 14110 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 14:54:06.996045 14116 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 14:54:06.995998 14118 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 14:54:06.996016 14110 server_base.cc:1034] running on GCE node
W20260927 14:54:06.996037 14115 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 14:54:06.996770 14110 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 14:54:06.997908 14110 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 14:54:06.999069 14110 hybrid_clock.cc:648] HybridClock initialized: now 1790520846999051 us; error 33 us; skew 500 ppm
Sep 27 14:54:07 dist-test-slave-xk1g krb5kdc[14097](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790520847, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 14:54:07.005726 14110 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 14:54:07.006649 14110 webserver.cc:460] Webserver started at http://127.0.98.62:42835/ using document root <none> and password file <none>
I20260927 14:54:07.007233 14110 fs_manager.cc:359] Metadata directory not provided
I20260927 14:54:07.007323 14110 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 14:54:07.007562 14110 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 14:54:07.009707 14110 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790520684602251-392-0/minicluster-data/master-0/data/instance:
uuid: "bbbbbbf5e7484aeea9277246d74d663b"
format_stamp: "Formatted at 2026-09-27 14:54:07 on dist-test-slave-xk1g"
server_key: "9cfcd137779d48b5a67105aae8a2a984"
server_key_iv: "6a588433d7e816f740f0b7dd7afcfe18"
server_key_version: "encryptionkey@0"
I20260927 14:54:07.010265 14110 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790520684602251-392-0/minicluster-data/master-0/wal/instance:
uuid: "bbbbbbf5e7484aeea9277246d74d663b"
format_stamp: "Formatted at 2026-09-27 14:54:07 on dist-test-slave-xk1g"
server_key: "9cfcd137779d48b5a67105aae8a2a984"
server_key_iv: "6a588433d7e816f740f0b7dd7afcfe18"
server_key_version: "encryptionkey@0"
I20260927 14:54:07.014075 14110 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260927 14:54:07.016422 14125 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 14:54:07.017674 14110 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 14:54:07.017812 14110 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790520684602251-392-0/minicluster-data/master-0/data,/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790520684602251-392-0/minicluster-data/master-0/wal
uuid: "bbbbbbf5e7484aeea9277246d74d663b"
format_stamp: "Formatted at 2026-09-27 14:54:07 on dist-test-slave-xk1g"
server_key: "9cfcd137779d48b5a67105aae8a2a984"
server_key_iv: "6a588433d7e816f740f0b7dd7afcfe18"
server_key_version: "encryptionkey@0"
I20260927 14:54:07.017925 14110 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790520684602251-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790520684602251-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790520684602251-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 14:54:07.026639 14110 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 14:54:07.029484 14110 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 14:54:07.029686 14110 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 14:54:07.036870 14110 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:40583
I20260927 14:54:07.037187 14177 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:40583 every 8 connection(s)
I20260927 14:54:07.037919 14110 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790520684602251-392-0/minicluster-data/master-0/data/info.pb
I20260927 14:54:07.040603 14178 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 14:54:07.041576 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6936m5/build/debug/bin/kudu as pid 14110
I20260927 14:54:07.041716 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790520684602251-392-0/minicluster-data/master-0/wal/instance
I20260927 14:54:07.042008 392 external_mini_cluster.cc:1428] Setting key b6d6fb1d5db7629f8c5b2f80c28883ae
I20260927 14:54:07.046335 14178 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bbbbbbf5e7484aeea9277246d74d663b: Bootstrap starting.
I20260927 14:54:07.048465 14178 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bbbbbbf5e7484aeea9277246d74d663b: Neither blocks nor log segments found. Creating new log.
I20260927 14:54:07.049242 14178 log.cc:824] T 00000000000000000000000000000000 P bbbbbbf5e7484aeea9277246d74d663b: Log is configured to *not* fsync() on all Append() calls
Sep 27 14:54:07 dist-test-slave-xk1g krb5kdc[14097](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790520846, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 14:54:07.051498 14178 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bbbbbbf5e7484aeea9277246d74d663b: No bootstrap required, opened a new log
I20260927 14:54:07.055485 14178 raft_consensus.cc:348] T 00000000000000000000000000000000 P bbbbbbf5e7484aeea9277246d74d663b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bbbbbbf5e7484aeea9277246d74d663b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40583 } }
I20260927 14:54:07.055706 14178 raft_consensus.cc:374] T 00000000000000000000000000000000 P bbbbbbf5e7484aeea9277246d74d663b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 14:54:07.055784 14178 raft_consensus.cc:729] T 00000000000000000000000000000000 P bbbbbbf5e7484aeea9277246d74d663b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bbbbbbf5e7484aeea9277246d74d663b, State: Initialized, Role: FOLLOWER
I20260927 14:54:07.056130 14181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:54:07.043322 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44636 (local address 127.0.98.62:40583)
0927 14:54:07.043772 (+ 450us) server_negotiation.cc:206] Beginning negotiation
0927 14:54:07.043788 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0927 14:54:07.043822 (+ 34us) server_negotiation.cc:407] Connection header received
0927 14:54:07.044425 (+ 603us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:54:07.044441 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:54:07.044768 (+ 327us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:54:07.045148 (+ 380us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:54:07.045921 (+ 773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:54:07.046926 (+ 1005us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:54:07.047775 (+ 849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:54:07.048246 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:54:07.048432 (+ 186us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:54:07.051351 (+ 2919us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:54:07.051356 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:54:07.051369 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 14:54:07.051394 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:54:07.053220 (+ 1826us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:54:07.053733 (+ 513us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:54:07.053738 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:54:07.053742 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:54:07.053845 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:54:07.054121 (+ 276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:54:07.054124 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:54:07.054125 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:54:07.054495 (+ 370us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:54:07.054662 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0927 14:54:07.054950 (+ 288us) server_negotiation.cc:299] Negotiation successful
0927 14:54:07.055278 (+ 328us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":303,"thread_start_us":134,"threads_started":1}
I20260927 14:54:07.056277 14178 consensus_queue.cc:260] T 00000000000000000000000000000000 P bbbbbbf5e7484aeea9277246d74d663b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bbbbbbf5e7484aeea9277246d74d663b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40583 } }
I20260927 14:54:07.056439 14178 raft_consensus.cc:388] T 00000000000000000000000000000000 P bbbbbbf5e7484aeea9277246d74d663b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 14:54:07.056509 14178 raft_consensus.cc:482] T 00000000000000000000000000000000 P bbbbbbf5e7484aeea9277246d74d663b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 14:54:07.056605 14178 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bbbbbbf5e7484aeea9277246d74d663b [term 0 FOLLOWER]: Advancing to term 1
I20260927 14:54:07.057564 14178 raft_consensus.cc:504] T 00000000000000000000000000000000 P bbbbbbf5e7484aeea9277246d74d663b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bbbbbbf5e7484aeea9277246d74d663b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40583 } }
I20260927 14:54:07.057905 14178 leader_election.cc:302] T 00000000000000000000000000000000 P bbbbbbf5e7484aeea9277246d74d663b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bbbbbbf5e7484aeea9277246d74d663b; no voters:
I20260927 14:54:07.058228 14178 leader_election.cc:288] T 00000000000000000000000000000000 P bbbbbbf5e7484aeea9277246d74d663b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 14:54:07.058308 14183 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bbbbbbf5e7484aeea9277246d74d663b [term 1 FOLLOWER]: Leader election won for term 1
I20260927 14:54:07.058578 14183 raft_consensus.cc:686] T 00000000000000000000000000000000 P bbbbbbf5e7484aeea9277246d74d663b [term 1 LEADER]: Becoming Leader. State: Replica: bbbbbbf5e7484aeea9277246d74d663b, State: Running, Role: LEADER
I20260927 14:54:07.058911 14183 consensus_queue.cc:237] T 00000000000000000000000000000000 P bbbbbbf5e7484aeea9277246d74d663b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bbbbbbf5e7484aeea9277246d74d663b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40583 } }
I20260927 14:54:07.059392 14178 sys_catalog.cc:564] T 00000000000000000000000000000000 P bbbbbbf5e7484aeea9277246d74d663b [sys.catalog]: configured and running, proceeding with master startup.
I20260927 14:54:07.060531 14185 sys_catalog.cc:455] T 00000000000000000000000000000000 P bbbbbbf5e7484aeea9277246d74d663b [sys.catalog]: SysCatalogTable state changed. Reason: New leader bbbbbbf5e7484aeea9277246d74d663b. Latest consensus state: current_term: 1 leader_uuid: "bbbbbbf5e7484aeea9277246d74d663b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bbbbbbf5e7484aeea9277246d74d663b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40583 } } }
I20260927 14:54:07.060663 14185 sys_catalog.cc:458] T 00000000000000000000000000000000 P bbbbbbf5e7484aeea9277246d74d663b [sys.catalog]: This master's current role is: LEADER
I20260927 14:54:07.060921 14184 sys_catalog.cc:455] T 00000000000000000000000000000000 P bbbbbbf5e7484aeea9277246d74d663b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bbbbbbf5e7484aeea9277246d74d663b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bbbbbbf5e7484aeea9277246d74d663b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40583 } } }
I20260927 14:54:07.061026 14184 sys_catalog.cc:458] T 00000000000000000000000000000000 P bbbbbbf5e7484aeea9277246d74d663b [sys.catalog]: This master's current role is: LEADER
I20260927 14:54:07.061121 14192 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 14:54:07.064144 14192 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 14:54:07.069738 14192 catalog_manager.cc:1347] Generated new cluster ID: 8c868f16380a4788ae11a60bca2797c4
I20260927 14:54:07.069857 14192 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 14:54:07.074329 14192 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 14:54:07.075245 14192 catalog_manager.cc:1504] Loading token signing keys...
I20260927 14:54:07.087232 14192 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bbbbbbf5e7484aeea9277246d74d663b: Generated new TSK 0
I20260927 14:54:07.088053 14192 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790520684602251-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790520684602251-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260927 14:54:07.120398 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6936m5/build/debug/bin/kudu
/tmp/dist-test-task6936m5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790520684602251-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790520684602251-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790520684602251-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790520684602251-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40583
--keytab_file=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790520684602251-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45035
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790520684602251-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790520684602251-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790520684602251-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 14:54:07.233506 14204 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 14:54:07.233770 14204 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 14:54:07.233844 14204 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 14:54:07.237918 14204 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 14:54:07.237978 14204 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 14:54:07.238035 14204 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 14:54:07.243198 14204 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45035
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790520684602251-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790520684602251-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790520684602251-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790520684602251-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40583
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14204
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790520684602251-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:15:17 UTC on ccycloud.quasar-ledxfe.root.comops.site
build id 4676134
I20260927 14:54:07.244297 14204 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 14:54:07.245373 14204 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 14:54:07.254925 14209 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 14:54:07.254937 14210 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 14:54:07.254966 14212 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 14:54:07.255228 14204 server_base.cc:1034] running on GCE node
I20260927 14:54:07.255599 14204 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 14:54:07.256259 14204 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 14:54:07.257448 14204 hybrid_clock.cc:648] HybridClock initialized: now 1790520847257433 us; error 27 us; skew 500 ppm
Sep 27 14:54:07 dist-test-slave-xk1g krb5kdc[14097](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790520847, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 14:54:07.260352 14204 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 14:54:07.261344 14204 webserver.cc:460] Webserver started at http://127.0.98.1:35933/ using document root <none> and password file <none>
I20260927 14:54:07.261861 14204 fs_manager.cc:359] Metadata directory not provided
I20260927 14:54:07.261916 14204 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 14:54:07.262075 14204 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 14:54:07.264014 14204 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790520684602251-392-0/minicluster-data/ts-0/data/instance:
uuid: "e3d6e62aa5b546dbaeadd8ba934aa24c"
format_stamp: "Formatted at 2026-09-27 14:54:07 on dist-test-slave-xk1g"
server_key: "a793d8b41e7bfe4a80c054b66217298c"
server_key_iv: "b9974b52713fa99a8ccf2c7ea4c775e1"
server_key_version: "encryptionkey@0"
I20260927 14:54:07.264505 14204 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790520684602251-392-0/minicluster-data/ts-0/wal/instance:
uuid: "e3d6e62aa5b546dbaeadd8ba934aa24c"
format_stamp: "Formatted at 2026-09-27 14:54:07 on dist-test-slave-xk1g"
server_key: "a793d8b41e7bfe4a80c054b66217298c"
server_key_iv: "b9974b52713fa99a8ccf2c7ea4c775e1"
server_key_version: "encryptionkey@0"
I20260927 14:54:07.268167 14204 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260927 14:54:07.270633 14219 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 14:54:07.271934 14204 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 14:54:07.272079 14204 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790520684602251-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790520684602251-392-0/minicluster-data/ts-0/wal
uuid: "e3d6e62aa5b546dbaeadd8ba934aa24c"
format_stamp: "Formatted at 2026-09-27 14:54:07 on dist-test-slave-xk1g"
server_key: "a793d8b41e7bfe4a80c054b66217298c"
server_key_iv: "b9974b52713fa99a8ccf2c7ea4c775e1"
server_key_version: "encryptionkey@0"
I20260927 14:54:07.272202 14204 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790520684602251-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790520684602251-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790520684602251-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 14:54:07.289793 14204 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 14:54:07.293040 14204 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 14:54:07.293310 14204 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 14:54:07.293939 14204 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 14:54:07.294831 14204 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 14:54:07.294909 14204 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 14:54:07.294986 14204 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 14:54:07.295037 14204 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 14:54:07.303930 14204 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:36587
I20260927 14:54:07.303953 14332 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:36587 every 8 connection(s)
I20260927 14:54:07.304879 14204 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790520684602251-392-0/minicluster-data/ts-0/data/info.pb
I20260927 14:54:07.306784 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6936m5/build/debug/bin/kudu as pid 14204
I20260927 14:54:07.306911 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790520684602251-392-0/minicluster-data/ts-0/wal/instance
I20260927 14:54:07.307255 392 external_mini_cluster.cc:1428] Setting key 8db9f29e3451d460aaea7e9c483d03a6
Sep 27 14:54:07 dist-test-slave-xk1g krb5kdc[14097](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790520847, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260927 14:54:07.321853 14181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:54:07.306947 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:52407 (local address 127.0.98.62:40583)
0927 14:54:07.307104 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0927 14:54:07.307109 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 14:54:07.307994 (+ 885us) server_negotiation.cc:407] Connection header received
0927 14:54:07.309714 (+ 1720us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:54:07.309717 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:54:07.309764 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:54:07.309834 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:54:07.311378 (+ 1544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:54:07.311931 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:54:07.313858 (+ 1927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:54:07.314196 (+ 338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:54:07.314302 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:54:07.317778 (+ 3476us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:54:07.317784 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:54:07.317789 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 14:54:07.317826 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:54:07.319408 (+ 1582us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:54:07.319948 (+ 540us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:54:07.319951 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:54:07.319952 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:54:07.320030 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:54:07.320447 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:54:07.320450 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:54:07.320452 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:54:07.320671 (+ 219us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:54:07.320819 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0927 14:54:07.321546 (+ 727us) server_negotiation.cc:299] Negotiation successful
0927 14:54:07.321687 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260927 14:54:07.322781 14335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:54:07.307256 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40583 (local address 127.0.98.1:52407)
0927 14:54:07.307787 (+ 531us) negotiation.cc:107] Waiting for socket to connect
0927 14:54:07.307868 (+ 81us) client_negotiation.cc:174] Beginning negotiation
0927 14:54:07.309459 (+ 1591us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:54:07.310068 (+ 609us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:54:07.310079 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:54:07.310651 (+ 572us) client_negotiation.cc:189] Negotiated authn=SASL
0927 14:54:07.311215 (+ 564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:54:07.311230 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:54:07.312073 (+ 843us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:54:07.312079 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:54:07.313681 (+ 1602us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:54:07.313706 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:54:07.314390 (+ 684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:54:07.314397 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:54:07.314656 (+ 259us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:54:07.315567 (+ 911us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 14:54:07.315596 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0927 14:54:07.317552 (+ 1956us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 14:54:07.319528 (+ 1976us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 14:54:07.319533 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 14:54:07.319548 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0927 14:54:07.319848 (+ 300us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 14:54:07.320117 (+ 269us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 14:54:07.320122 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 14:54:07.320128 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0927 14:54:07.320310 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 14:54:07.320851 (+ 541us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 14:54:07.320860 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 14:54:07.321276 (+ 416us) client_negotiation.cc:769] Sending connection context
0927 14:54:07.321486 (+ 210us) client_negotiation.cc:240] Negotiation successful
0927 14:54:07.321865 (+ 379us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":332,"thread_start_us":109,"threads_started":1}
I20260927 14:54:07.324316 14333 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40583
I20260927 14:54:07.324596 14333 heartbeater.cc:464] Registering TS with master...
I20260927 14:54:07.325277 14333 heartbeater.cc:511] Master 127.0.98.62:40583 requested a full tablet report, sending...
I20260927 14:54:07.327240 14142 ts_manager.cc:194] Registered new tserver with Master: e3d6e62aa5b546dbaeadd8ba934aa24c (127.0.98.1:36587)
I20260927 14:54:07.328928 14142 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:52407
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790520684602251-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790520684602251-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260927 14:54:07.340106 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6936m5/build/debug/bin/kudu
/tmp/dist-test-task6936m5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790520684602251-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790520684602251-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790520684602251-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790520684602251-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40583
--keytab_file=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790520684602251-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45035
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790520684602251-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790520684602251-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790520684602251-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 14:54:07.452013 14338 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 14:54:07.452258 14338 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 14:54:07.452302 14338 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 14:54:07.456249 14338 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 14:54:07.456315 14338 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 14:54:07.456374 14338 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 14:54:07.461748 14338 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45035
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790520684602251-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790520684602251-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790520684602251-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790520684602251-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40583
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14338
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790520684602251-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:15:17 UTC on ccycloud.quasar-ledxfe.root.comops.site
build id 4676134
I20260927 14:54:07.462721 14338 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 14:54:07.463673 14338 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 14:54:07.472892 14343 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 14:54:07.472899 14346 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 14:54:07.473001 14338 server_base.cc:1034] running on GCE node
W20260927 14:54:07.472903 14344 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 14:54:07.473582 14338 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 14:54:07.474265 14338 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 14:54:07.475466 14338 hybrid_clock.cc:648] HybridClock initialized: now 1790520847475450 us; error 27 us; skew 500 ppm
Sep 27 14:54:07 dist-test-slave-xk1g krb5kdc[14097](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790520847, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 14:54:07.478366 14338 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 14:54:07.479235 14338 webserver.cc:460] Webserver started at http://127.0.98.2:45923/ using document root <none> and password file <none>
I20260927 14:54:07.479799 14338 fs_manager.cc:359] Metadata directory not provided
I20260927 14:54:07.479889 14338 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 14:54:07.480131 14338 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 14:54:07.482209 14338 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790520684602251-392-0/minicluster-data/ts-1/data/instance:
uuid: "148385836b554faaba8ee69765e73730"
format_stamp: "Formatted at 2026-09-27 14:54:07 on dist-test-slave-xk1g"
server_key: "bb1f0ed822f3151730bc37dd3e4db41f"
server_key_iv: "5b105ad590e22475019065d8d3694582"
server_key_version: "encryptionkey@0"
I20260927 14:54:07.482769 14338 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790520684602251-392-0/minicluster-data/ts-1/wal/instance:
uuid: "148385836b554faaba8ee69765e73730"
format_stamp: "Formatted at 2026-09-27 14:54:07 on dist-test-slave-xk1g"
server_key: "bb1f0ed822f3151730bc37dd3e4db41f"
server_key_iv: "5b105ad590e22475019065d8d3694582"
server_key_version: "encryptionkey@0"
I20260927 14:54:07.486461 14338 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 14:54:07.488720 14353 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 14:54:07.489905 14338 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 14:54:07.490041 14338 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790520684602251-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790520684602251-392-0/minicluster-data/ts-1/wal
uuid: "148385836b554faaba8ee69765e73730"
format_stamp: "Formatted at 2026-09-27 14:54:07 on dist-test-slave-xk1g"
server_key: "bb1f0ed822f3151730bc37dd3e4db41f"
server_key_iv: "5b105ad590e22475019065d8d3694582"
server_key_version: "encryptionkey@0"
I20260927 14:54:07.490171 14338 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790520684602251-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790520684602251-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790520684602251-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 14:54:07.503463 14338 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 14:54:07.506482 14338 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 14:54:07.506716 14338 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 14:54:07.507310 14338 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 14:54:07.508167 14338 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 14:54:07.508239 14338 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 14:54:07.508309 14338 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 14:54:07.508450 14338 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 14:54:07.517997 14338 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:32885
I20260927 14:54:07.518019 14466 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:32885 every 8 connection(s)
I20260927 14:54:07.518975 14338 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790520684602251-392-0/minicluster-data/ts-1/data/info.pb
I20260927 14:54:07.526283 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6936m5/build/debug/bin/kudu as pid 14338
I20260927 14:54:07.526420 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790520684602251-392-0/minicluster-data/ts-1/wal/instance
I20260927 14:54:07.526698 392 external_mini_cluster.cc:1428] Setting key 913524f208d93f3d1a961df714679e35
Sep 27 14:54:07 dist-test-slave-xk1g krb5kdc[14097](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790520847, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 14:54:07.533571 14181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:54:07.520850 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:38271 (local address 127.0.98.62:40583)
0927 14:54:07.521017 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0927 14:54:07.521021 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 14:54:07.521947 (+ 926us) server_negotiation.cc:407] Connection header received
0927 14:54:07.523275 (+ 1328us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:54:07.523278 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:54:07.523326 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:54:07.523403 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:54:07.524610 (+ 1207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:54:07.525201 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:54:07.526415 (+ 1214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:54:07.526730 (+ 315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:54:07.526861 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:54:07.529593 (+ 2732us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:54:07.529597 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:54:07.529599 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 14:54:07.529624 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:54:07.531128 (+ 1504us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:54:07.531679 (+ 551us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:54:07.531683 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:54:07.531685 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:54:07.531800 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:54:07.532236 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:54:07.532242 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:54:07.532247 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:54:07.532511 (+ 264us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:54:07.532626 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0927 14:54:07.533177 (+ 551us) server_negotiation.cc:299] Negotiation successful
0927 14:54:07.533333 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260927 14:54:07.534503 14469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:54:07.521323 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40583 (local address 127.0.98.2:38271)
0927 14:54:07.521792 (+ 469us) negotiation.cc:107] Waiting for socket to connect
0927 14:54:07.521837 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0927 14:54:07.523081 (+ 1244us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:54:07.523589 (+ 508us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:54:07.523597 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:54:07.523985 (+ 388us) client_negotiation.cc:189] Negotiated authn=SASL
0927 14:54:07.524473 (+ 488us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:54:07.524486 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:54:07.525339 (+ 853us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:54:07.525343 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:54:07.526264 (+ 921us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:54:07.526274 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:54:07.526920 (+ 646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:54:07.526925 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:54:07.527138 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:54:07.527798 (+ 660us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 14:54:07.527815 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 14:54:07.529414 (+ 1599us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 14:54:07.531265 (+ 1851us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 14:54:07.531271 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 14:54:07.531287 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0927 14:54:07.531571 (+ 284us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 14:54:07.531910 (+ 339us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 14:54:07.531913 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 14:54:07.531915 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 14:54:07.532128 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 14:54:07.532626 (+ 498us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 14:54:07.532631 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 14:54:07.532876 (+ 245us) client_negotiation.cc:769] Sending connection context
0927 14:54:07.533155 (+ 279us) client_negotiation.cc:240] Negotiation successful
0927 14:54:07.533492 (+ 337us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":269,"thread_start_us":104,"threads_started":1}
I20260927 14:54:07.535838 14467 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40583
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260927 14:54:07.536128 14467 heartbeater.cc:464] Registering TS with master...
I20260927 14:54:07.536672 14467 heartbeater.cc:511] Master 127.0.98.62:40583 requested a full tablet report, sending...
I20260927 14:54:07.537858 14142 ts_manager.cc:194] Registered new tserver with Master: 148385836b554faaba8ee69765e73730 (127.0.98.2:32885)
I20260927 14:54:07.538494 14142 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:38271
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790520684602251-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790520684602251-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260927 14:54:07.556223 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6936m5/build/debug/bin/kudu
/tmp/dist-test-task6936m5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790520684602251-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790520684602251-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790520684602251-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790520684602251-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40583
--keytab_file=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790520684602251-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45035
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790520684602251-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790520684602251-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790520684602251-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 14:54:07.668917 14472 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 14:54:07.669306 14472 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 14:54:07.669369 14472 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 14:54:07.673388 14472 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 14:54:07.673442 14472 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 14:54:07.673502 14472 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 14:54:07.678802 14472 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45035
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790520684602251-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790520684602251-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790520684602251-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790520684602251-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40583
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14472
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790520684602251-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:15:17 UTC on ccycloud.quasar-ledxfe.root.comops.site
build id 4676134
I20260927 14:54:07.679788 14472 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 14:54:07.680711 14472 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 14:54:07.690187 14480 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 14:54:07.690189 14477 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 14:54:07.690187 14478 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 14:54:07.690521 14472 server_base.cc:1034] running on GCE node
I20260927 14:54:07.690950 14472 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 14:54:07.691571 14472 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 14:54:07.692770 14472 hybrid_clock.cc:648] HybridClock initialized: now 1790520847692737 us; error 45 us; skew 500 ppm
Sep 27 14:54:07 dist-test-slave-xk1g krb5kdc[14097](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790520847, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 14:54:07.695538 14472 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 14:54:07.696427 14472 webserver.cc:460] Webserver started at http://127.0.98.3:46651/ using document root <none> and password file <none>
I20260927 14:54:07.696979 14472 fs_manager.cc:359] Metadata directory not provided
I20260927 14:54:07.697067 14472 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 14:54:07.697322 14472 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 14:54:07.699290 14472 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790520684602251-392-0/minicluster-data/ts-2/data/instance:
uuid: "20705812c28647ec9ef08248d5de0736"
format_stamp: "Formatted at 2026-09-27 14:54:07 on dist-test-slave-xk1g"
server_key: "b8a0672c73a73a48a6e0652b1e6efab0"
server_key_iv: "b8386891c67ea1e1632665d2d00fb3dd"
server_key_version: "encryptionkey@0"
I20260927 14:54:07.699827 14472 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790520684602251-392-0/minicluster-data/ts-2/wal/instance:
uuid: "20705812c28647ec9ef08248d5de0736"
format_stamp: "Formatted at 2026-09-27 14:54:07 on dist-test-slave-xk1g"
server_key: "b8a0672c73a73a48a6e0652b1e6efab0"
server_key_iv: "b8386891c67ea1e1632665d2d00fb3dd"
server_key_version: "encryptionkey@0"
I20260927 14:54:07.703420 14472 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260927 14:54:07.705816 14487 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 14:54:07.706938 14472 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 14:54:07.707068 14472 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790520684602251-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790520684602251-392-0/minicluster-data/ts-2/wal
uuid: "20705812c28647ec9ef08248d5de0736"
format_stamp: "Formatted at 2026-09-27 14:54:07 on dist-test-slave-xk1g"
server_key: "b8a0672c73a73a48a6e0652b1e6efab0"
server_key_iv: "b8386891c67ea1e1632665d2d00fb3dd"
server_key_version: "encryptionkey@0"
I20260927 14:54:07.707173 14472 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790520684602251-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790520684602251-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790520684602251-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 14:54:07.728827 14472 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 14:54:07.731508 14472 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 14:54:07.731711 14472 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 14:54:07.732318 14472 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 14:54:07.733179 14472 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 14:54:07.733227 14472 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 14:54:07.733268 14472 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 14:54:07.733289 14472 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 14:54:07.742038 14472 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:42807
I20260927 14:54:07.742069 14600 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:42807 every 8 connection(s)
I20260927 14:54:07.742908 14472 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790520684602251-392-0/minicluster-data/ts-2/data/info.pb
Sep 27 14:54:07 dist-test-slave-xk1g krb5kdc[14097](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790520847, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 14:54:07.752583 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6936m5/build/debug/bin/kudu as pid 14472
I20260927 14:54:07.752717 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790520684602251-392-0/minicluster-data/ts-2/wal/instance
I20260927 14:54:07.752976 392 external_mini_cluster.cc:1428] Setting key 928a4d06598d10628cca4f013444d09a
I20260927 14:54:07.755748 14181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:54:07.744634 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:47901 (local address 127.0.98.62:40583)
0927 14:54:07.744797 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0927 14:54:07.744802 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 14:54:07.745647 (+ 845us) server_negotiation.cc:407] Connection header received
0927 14:54:07.746841 (+ 1194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:54:07.746844 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:54:07.746889 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:54:07.746961 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:54:07.748059 (+ 1098us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:54:07.748575 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:54:07.749633 (+ 1058us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:54:07.749841 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:54:07.749919 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:54:07.752138 (+ 2219us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:54:07.752141 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:54:07.752143 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 14:54:07.752165 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:54:07.753784 (+ 1619us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:54:07.754318 (+ 534us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:54:07.754325 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:54:07.754327 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:54:07.754412 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:54:07.754752 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:54:07.754755 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:54:07.754757 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:54:07.754972 (+ 215us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:54:07.755069 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0927 14:54:07.755527 (+ 458us) server_negotiation.cc:299] Negotiation successful
0927 14:54:07.755634 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260927 14:54:07.756497 14603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:54:07.745030 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40583 (local address 127.0.98.3:47901)
0927 14:54:07.745502 (+ 472us) negotiation.cc:107] Waiting for socket to connect
0927 14:54:07.745544 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0927 14:54:07.746654 (+ 1110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:54:07.747121 (+ 467us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:54:07.747130 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:54:07.747470 (+ 340us) client_negotiation.cc:189] Negotiated authn=SASL
0927 14:54:07.747941 (+ 471us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:54:07.747954 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:54:07.748679 (+ 725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:54:07.748681 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:54:07.749533 (+ 852us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:54:07.749542 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:54:07.749964 (+ 422us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:54:07.749967 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:54:07.750100 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:54:07.750612 (+ 512us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 14:54:07.750638 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0927 14:54:07.751984 (+ 1346us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 14:54:07.753908 (+ 1924us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 14:54:07.753915 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 14:54:07.753929 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0927 14:54:07.754224 (+ 295us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 14:54:07.754524 (+ 300us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 14:54:07.754526 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 14:54:07.754528 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 14:54:07.754654 (+ 126us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 14:54:07.755080 (+ 426us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 14:54:07.755086 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 14:54:07.755320 (+ 234us) client_negotiation.cc:769] Sending connection context
0927 14:54:07.755474 (+ 154us) client_negotiation.cc:240] Negotiation successful
0927 14:54:07.755720 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":250,"thread_start_us":101,"threads_started":1}
I20260927 14:54:07.757639 14601 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40583
I20260927 14:54:07.757829 14601 heartbeater.cc:464] Registering TS with master...
I20260927 14:54:07.758240 14601 heartbeater.cc:511] Master 127.0.98.62:40583 requested a full tablet report, sending...
I20260927 14:54:07.759183 14142 ts_manager.cc:194] Registered new tserver with Master: 20705812c28647ec9ef08248d5de0736 (127.0.98.3:42807)
I20260927 14:54:07.759691 14142 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:47901
I20260927 14:54:07.766872 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 14:54:07.775846 14181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:54:07.768749 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56838 (local address 127.0.98.62:40583)
0927 14:54:07.768883 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0927 14:54:07.768888 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 14:54:07.769109 (+ 221us) server_negotiation.cc:407] Connection header received
0927 14:54:07.769287 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:54:07.769289 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:54:07.769329 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:54:07.769411 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:54:07.770255 (+ 844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:54:07.770795 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:54:07.771651 (+ 856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:54:07.771862 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:54:07.771933 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:54:07.773265 (+ 1332us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:54:07.773268 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:54:07.773270 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 14:54:07.773291 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:54:07.774491 (+ 1200us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:54:07.774850 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:54:07.774852 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:54:07.774854 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:54:07.774921 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:54:07.775158 (+ 237us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:54:07.775160 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:54:07.775161 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:54:07.775346 (+ 185us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:54:07.775442 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0927 14:54:07.775642 (+ 200us) server_negotiation.cc:299] Negotiation successful
0927 14:54:07.775734 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260927 14:54:07.778426 14142 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56838:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 14:54:07.781314 14142 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 14:54:07.795135 14616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:54:07.789816 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:32885 (local address 127.0.0.1:38686)
0927 14:54:07.790327 (+ 511us) negotiation.cc:107] Waiting for socket to connect
0927 14:54:07.790340 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 14:54:07.790432 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:54:07.791071 (+ 639us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:54:07.791073 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:54:07.791095 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 14:54:07.791185 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:54:07.791191 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:54:07.792185 (+ 994us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:54:07.792191 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:54:07.793537 (+ 1346us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:54:07.793547 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:54:07.794564 (+ 1017us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:54:07.794569 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:54:07.794681 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:54:07.794735 (+ 54us) client_negotiation.cc:769] Sending connection context
0927 14:54:07.794837 (+ 102us) client_negotiation.cc:240] Negotiation successful
0927 14:54:07.794952 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":451,"mutex_wait_us":69,"thread_start_us":96,"threads_started":1}
I20260927 14:54:07.795389 14613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:54:07.789122 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:42807 (local address 127.0.0.1:58538)
0927 14:54:07.789882 (+ 760us) negotiation.cc:107] Waiting for socket to connect
0927 14:54:07.789931 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0927 14:54:07.790086 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:54:07.790764 (+ 678us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:54:07.790769 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:54:07.790795 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 14:54:07.790923 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:54:07.790929 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:54:07.792332 (+ 1403us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:54:07.792335 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:54:07.793806 (+ 1471us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:54:07.793817 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:54:07.794860 (+ 1043us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:54:07.794864 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:54:07.794951 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:54:07.794985 (+ 34us) client_negotiation.cc:769] Sending connection context
0927 14:54:07.795062 (+ 77us) client_negotiation.cc:240] Negotiation successful
0927 14:54:07.795129 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":599,"mutex_wait_us":35,"thread_start_us":65,"threads_started":1}
I20260927 14:54:07.795564 14614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:54:07.789260 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58538 (local address 127.0.98.3:42807)
0927 14:54:07.789628 (+ 368us) server_negotiation.cc:206] Beginning negotiation
0927 14:54:07.789640 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 14:54:07.790239 (+ 599us) server_negotiation.cc:407] Connection header received
0927 14:54:07.790322 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:54:07.790327 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:54:07.790539 (+ 212us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:54:07.790735 (+ 196us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 14:54:07.791035 (+ 300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:54:07.792187 (+ 1152us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:54:07.793929 (+ 1742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:54:07.794756 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:54:07.794861 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:54:07.794987 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0927 14:54:07.795100 (+ 113us) server_negotiation.cc:299] Negotiation successful
0927 14:54:07.795212 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":272,"thread_start_us":191,"threads_started":1}
I20260927 14:54:07.795674 14617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:54:07.789978 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38686 (local address 127.0.98.2:32885)
0927 14:54:07.790358 (+ 380us) server_negotiation.cc:206] Beginning negotiation
0927 14:54:07.790373 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 14:54:07.790397 (+ 24us) server_negotiation.cc:407] Connection header received
0927 14:54:07.790742 (+ 345us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:54:07.790748 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:54:07.790924 (+ 176us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:54:07.791072 (+ 148us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 14:54:07.791276 (+ 204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:54:07.792056 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:54:07.793663 (+ 1607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:54:07.794442 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:54:07.794547 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:54:07.794696 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0927 14:54:07.795338 (+ 642us) server_negotiation.cc:299] Negotiation successful
0927 14:54:07.795429 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":292,"thread_start_us":98,"threads_started":1}
I20260927 14:54:07.795894 14612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:54:07.789296 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:36587 (local address 127.0.0.1:59788)
0927 14:54:07.790137 (+ 841us) negotiation.cc:107] Waiting for socket to connect
0927 14:54:07.790153 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 14:54:07.790494 (+ 341us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:54:07.791242 (+ 748us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:54:07.791247 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:54:07.791265 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 14:54:07.791372 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:54:07.791378 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:54:07.792744 (+ 1366us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:54:07.792748 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:54:07.794014 (+ 1266us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:54:07.794023 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:54:07.795407 (+ 1384us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:54:07.795411 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:54:07.795597 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:54:07.795618 (+ 21us) client_negotiation.cc:769] Sending connection context
0927 14:54:07.795695 (+ 77us) client_negotiation.cc:240] Negotiation successful
0927 14:54:07.795769 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":777,"mutex_wait_us":19,"thread_start_us":77,"threads_started":1}
I20260927 14:54:07.796337 14615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:54:07.789800 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59788 (local address 127.0.98.1:36587)
0927 14:54:07.790686 (+ 886us) server_negotiation.cc:206] Beginning negotiation
0927 14:54:07.790707 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0927 14:54:07.790728 (+ 21us) server_negotiation.cc:407] Connection header received
0927 14:54:07.790793 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:54:07.790797 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:54:07.790940 (+ 143us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:54:07.791072 (+ 132us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 14:54:07.791508 (+ 436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:54:07.792596 (+ 1088us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:54:07.794131 (+ 1535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:54:07.794938 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:54:07.795045 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:54:07.795191 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0927 14:54:07.796111 (+ 920us) server_negotiation.cc:299] Negotiation successful
0927 14:54:07.796201 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":773,"thread_start_us":85,"threads_started":1}
I20260927 14:54:07.798553 14401 tablet_service.cc:1467] Processing CreateTablet for tablet 44650e4deef944c2a35f672a8b0b824e (DEFAULT_TABLE table=test-table [id=58d23e136bf84ea09cfd849e6e3496c6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 14:54:07.798542 14535 tablet_service.cc:1467] Processing CreateTablet for tablet 44650e4deef944c2a35f672a8b0b824e (DEFAULT_TABLE table=test-table [id=58d23e136bf84ea09cfd849e6e3496c6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 14:54:07.798745 14267 tablet_service.cc:1467] Processing CreateTablet for tablet 44650e4deef944c2a35f672a8b0b824e (DEFAULT_TABLE table=test-table [id=58d23e136bf84ea09cfd849e6e3496c6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 14:54:07.799407 14401 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 44650e4deef944c2a35f672a8b0b824e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 14:54:07.799511 14267 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 44650e4deef944c2a35f672a8b0b824e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 14:54:07.799518 14535 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 44650e4deef944c2a35f672a8b0b824e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 14:54:07.805891 14618 tablet_bootstrap.cc:492] T 44650e4deef944c2a35f672a8b0b824e P 148385836b554faaba8ee69765e73730: Bootstrap starting.
I20260927 14:54:07.806238 14619 tablet_bootstrap.cc:492] T 44650e4deef944c2a35f672a8b0b824e P 20705812c28647ec9ef08248d5de0736: Bootstrap starting.
I20260927 14:54:07.806988 14620 tablet_bootstrap.cc:492] T 44650e4deef944c2a35f672a8b0b824e P e3d6e62aa5b546dbaeadd8ba934aa24c: Bootstrap starting.
I20260927 14:54:07.808357 14619 tablet_bootstrap.cc:654] T 44650e4deef944c2a35f672a8b0b824e P 20705812c28647ec9ef08248d5de0736: Neither blocks nor log segments found. Creating new log.
I20260927 14:54:07.808779 14618 tablet_bootstrap.cc:654] T 44650e4deef944c2a35f672a8b0b824e P 148385836b554faaba8ee69765e73730: Neither blocks nor log segments found. Creating new log.
I20260927 14:54:07.808818 14620 tablet_bootstrap.cc:654] T 44650e4deef944c2a35f672a8b0b824e P e3d6e62aa5b546dbaeadd8ba934aa24c: Neither blocks nor log segments found. Creating new log.
I20260927 14:54:07.809731 14619 log.cc:824] T 44650e4deef944c2a35f672a8b0b824e P 20705812c28647ec9ef08248d5de0736: Log is configured to *not* fsync() on all Append() calls
I20260927 14:54:07.809731 14618 log.cc:824] T 44650e4deef944c2a35f672a8b0b824e P 148385836b554faaba8ee69765e73730: Log is configured to *not* fsync() on all Append() calls
I20260927 14:54:07.809841 14620 log.cc:824] T 44650e4deef944c2a35f672a8b0b824e P e3d6e62aa5b546dbaeadd8ba934aa24c: Log is configured to *not* fsync() on all Append() calls
I20260927 14:54:07.811841 14618 tablet_bootstrap.cc:492] T 44650e4deef944c2a35f672a8b0b824e P 148385836b554faaba8ee69765e73730: No bootstrap required, opened a new log
I20260927 14:54:07.811863 14620 tablet_bootstrap.cc:492] T 44650e4deef944c2a35f672a8b0b824e P e3d6e62aa5b546dbaeadd8ba934aa24c: No bootstrap required, opened a new log
I20260927 14:54:07.811867 14619 tablet_bootstrap.cc:492] T 44650e4deef944c2a35f672a8b0b824e P 20705812c28647ec9ef08248d5de0736: No bootstrap required, opened a new log
I20260927 14:54:07.812044 14618 ts_tablet_manager.cc:1397] T 44650e4deef944c2a35f672a8b0b824e P 148385836b554faaba8ee69765e73730: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.003s
I20260927 14:54:07.812062 14619 ts_tablet_manager.cc:1397] T 44650e4deef944c2a35f672a8b0b824e P 20705812c28647ec9ef08248d5de0736: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260927 14:54:07.812072 14620 ts_tablet_manager.cc:1397] T 44650e4deef944c2a35f672a8b0b824e P e3d6e62aa5b546dbaeadd8ba934aa24c: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260927 14:54:07.814839 14618 raft_consensus.cc:348] T 44650e4deef944c2a35f672a8b0b824e P 148385836b554faaba8ee69765e73730 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20705812c28647ec9ef08248d5de0736" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42807 } } peers { permanent_uuid: "e3d6e62aa5b546dbaeadd8ba934aa24c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36587 } } peers { permanent_uuid: "148385836b554faaba8ee69765e73730" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32885 } }
I20260927 14:54:07.815079 14618 raft_consensus.cc:374] T 44650e4deef944c2a35f672a8b0b824e P 148385836b554faaba8ee69765e73730 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 14:54:07.815138 14618 raft_consensus.cc:729] T 44650e4deef944c2a35f672a8b0b824e P 148385836b554faaba8ee69765e73730 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 148385836b554faaba8ee69765e73730, State: Initialized, Role: FOLLOWER
I20260927 14:54:07.815562 14618 consensus_queue.cc:260] T 44650e4deef944c2a35f672a8b0b824e P 148385836b554faaba8ee69765e73730 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20705812c28647ec9ef08248d5de0736" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42807 } } peers { permanent_uuid: "e3d6e62aa5b546dbaeadd8ba934aa24c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36587 } } peers { permanent_uuid: "148385836b554faaba8ee69765e73730" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32885 } }
I20260927 14:54:07.815852 14619 raft_consensus.cc:348] T 44650e4deef944c2a35f672a8b0b824e P 20705812c28647ec9ef08248d5de0736 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20705812c28647ec9ef08248d5de0736" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42807 } } peers { permanent_uuid: "e3d6e62aa5b546dbaeadd8ba934aa24c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36587 } } peers { permanent_uuid: "148385836b554faaba8ee69765e73730" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32885 } }
I20260927 14:54:07.815852 14620 raft_consensus.cc:348] T 44650e4deef944c2a35f672a8b0b824e P e3d6e62aa5b546dbaeadd8ba934aa24c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20705812c28647ec9ef08248d5de0736" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42807 } } peers { permanent_uuid: "e3d6e62aa5b546dbaeadd8ba934aa24c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36587 } } peers { permanent_uuid: "148385836b554faaba8ee69765e73730" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32885 } }
I20260927 14:54:07.816102 14619 raft_consensus.cc:374] T 44650e4deef944c2a35f672a8b0b824e P 20705812c28647ec9ef08248d5de0736 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 14:54:07.816102 14620 raft_consensus.cc:374] T 44650e4deef944c2a35f672a8b0b824e P e3d6e62aa5b546dbaeadd8ba934aa24c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 14:54:07.816164 14619 raft_consensus.cc:729] T 44650e4deef944c2a35f672a8b0b824e P 20705812c28647ec9ef08248d5de0736 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 20705812c28647ec9ef08248d5de0736, State: Initialized, Role: FOLLOWER
I20260927 14:54:07.816164 14620 raft_consensus.cc:729] T 44650e4deef944c2a35f672a8b0b824e P e3d6e62aa5b546dbaeadd8ba934aa24c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e3d6e62aa5b546dbaeadd8ba934aa24c, State: Initialized, Role: FOLLOWER
I20260927 14:54:07.816295 14467 heartbeater.cc:503] Master 127.0.98.62:40583 was elected leader, sending a full tablet report...
I20260927 14:54:07.816534 14618 ts_tablet_manager.cc:1428] T 44650e4deef944c2a35f672a8b0b824e P 148385836b554faaba8ee69765e73730: Time spent starting tablet: real 0.004s user 0.003s sys 0.002s
I20260927 14:54:07.816608 14619 consensus_queue.cc:260] T 44650e4deef944c2a35f672a8b0b824e P 20705812c28647ec9ef08248d5de0736 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20705812c28647ec9ef08248d5de0736" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42807 } } peers { permanent_uuid: "e3d6e62aa5b546dbaeadd8ba934aa24c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36587 } } peers { permanent_uuid: "148385836b554faaba8ee69765e73730" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32885 } }
I20260927 14:54:07.816608 14620 consensus_queue.cc:260] T 44650e4deef944c2a35f672a8b0b824e P e3d6e62aa5b546dbaeadd8ba934aa24c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20705812c28647ec9ef08248d5de0736" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42807 } } peers { permanent_uuid: "e3d6e62aa5b546dbaeadd8ba934aa24c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36587 } } peers { permanent_uuid: "148385836b554faaba8ee69765e73730" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32885 } }
I20260927 14:54:07.817449 14333 heartbeater.cc:503] Master 127.0.98.62:40583 was elected leader, sending a full tablet report...
I20260927 14:54:07.817495 14601 heartbeater.cc:503] Master 127.0.98.62:40583 was elected leader, sending a full tablet report...
I20260927 14:54:07.819016 14620 ts_tablet_manager.cc:1428] T 44650e4deef944c2a35f672a8b0b824e P e3d6e62aa5b546dbaeadd8ba934aa24c: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260927 14:54:07.819614 14619 ts_tablet_manager.cc:1428] T 44650e4deef944c2a35f672a8b0b824e P 20705812c28647ec9ef08248d5de0736: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260927 14:54:07.916994 14625 raft_consensus.cc:482] T 44650e4deef944c2a35f672a8b0b824e P e3d6e62aa5b546dbaeadd8ba934aa24c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 14:54:07.917269 14625 raft_consensus.cc:504] T 44650e4deef944c2a35f672a8b0b824e P e3d6e62aa5b546dbaeadd8ba934aa24c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20705812c28647ec9ef08248d5de0736" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42807 } } peers { permanent_uuid: "e3d6e62aa5b546dbaeadd8ba934aa24c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36587 } } peers { permanent_uuid: "148385836b554faaba8ee69765e73730" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32885 } }
I20260927 14:54:07.918392 14625 leader_election.cc:288] T 44650e4deef944c2a35f672a8b0b824e P e3d6e62aa5b546dbaeadd8ba934aa24c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 20705812c28647ec9ef08248d5de0736 (127.0.98.3:42807), 148385836b554faaba8ee69765e73730 (127.0.98.2:32885)
I20260927 14:54:07.922783 14617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:54:07.918796 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:37103 (local address 127.0.98.2:32885)
0927 14:54:07.919050 (+ 254us) server_negotiation.cc:206] Beginning negotiation
0927 14:54:07.919057 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 14:54:07.919377 (+ 320us) server_negotiation.cc:407] Connection header received
0927 14:54:07.919508 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:54:07.919510 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:54:07.919554 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:54:07.919641 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 14:54:07.919913 (+ 272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:54:07.920398 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:54:07.921734 (+ 1336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:54:07.922235 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:54:07.922352 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:54:07.922414 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0927 14:54:07.922595 (+ 181us) server_negotiation.cc:299] Negotiation successful
0927 14:54:07.922662 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260927 14:54:07.922801 14628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:54:07.918730 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:32885 (local address 127.0.98.1:37103)
0927 14:54:07.919273 (+ 543us) negotiation.cc:107] Waiting for socket to connect
0927 14:54:07.919284 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 14:54:07.919406 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:54:07.919647 (+ 241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:54:07.919650 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:54:07.919666 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 14:54:07.919800 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:54:07.919805 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:54:07.920506 (+ 701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:54:07.920513 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:54:07.921598 (+ 1085us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:54:07.921611 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:54:07.922355 (+ 744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:54:07.922359 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:54:07.922462 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:54:07.922491 (+ 29us) client_negotiation.cc:769] Sending connection context
0927 14:54:07.922578 (+ 87us) client_negotiation.cc:240] Negotiation successful
0927 14:54:07.922647 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":484,"mutex_wait_us":96,"thread_start_us":134,"threads_started":1}
I20260927 14:54:07.922911 14614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:54:07.918844 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:47751 (local address 127.0.98.3:42807)
0927 14:54:07.919106 (+ 262us) server_negotiation.cc:206] Beginning negotiation
0927 14:54:07.919113 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 14:54:07.919139 (+ 26us) server_negotiation.cc:407] Connection header received
0927 14:54:07.919343 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:54:07.919347 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:54:07.919408 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:54:07.919505 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 14:54:07.919826 (+ 321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:54:07.920349 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:54:07.921735 (+ 1386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:54:07.922280 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:54:07.922405 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:54:07.922479 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0927 14:54:07.922693 (+ 214us) server_negotiation.cc:299] Negotiation successful
0927 14:54:07.922765 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":136}
I20260927 14:54:07.923187 14627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:54:07.918717 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:42807 (local address 127.0.98.1:47751)
0927 14:54:07.919058 (+ 341us) negotiation.cc:107] Waiting for socket to connect
0927 14:54:07.919076 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 14:54:07.919201 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:54:07.919540 (+ 339us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:54:07.919543 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:54:07.919561 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 14:54:07.919701 (+ 140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:54:07.919707 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:54:07.920531 (+ 824us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:54:07.920534 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:54:07.921607 (+ 1073us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:54:07.921617 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:54:07.922418 (+ 801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:54:07.922422 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:54:07.922554 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:54:07.922583 (+ 29us) client_negotiation.cc:769] Sending connection context
0927 14:54:07.922677 (+ 94us) client_negotiation.cc:240] Negotiation successful
0927 14:54:07.922748 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":256,"mutex_wait_us":22,"thread_start_us":100,"threads_started":1}
I20260927 14:54:07.923400 14421 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44650e4deef944c2a35f672a8b0b824e" candidate_uuid: "e3d6e62aa5b546dbaeadd8ba934aa24c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "148385836b554faaba8ee69765e73730" is_pre_election: true
I20260927 14:54:07.923746 14421 raft_consensus.cc:2454] T 44650e4deef944c2a35f672a8b0b824e P 148385836b554faaba8ee69765e73730 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e3d6e62aa5b546dbaeadd8ba934aa24c in term 0.
I20260927 14:54:07.923889 14555 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44650e4deef944c2a35f672a8b0b824e" candidate_uuid: "e3d6e62aa5b546dbaeadd8ba934aa24c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "20705812c28647ec9ef08248d5de0736" is_pre_election: true
I20260927 14:54:07.924221 14555 raft_consensus.cc:2454] T 44650e4deef944c2a35f672a8b0b824e P 20705812c28647ec9ef08248d5de0736 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e3d6e62aa5b546dbaeadd8ba934aa24c in term 0.
I20260927 14:54:07.924232 14220 leader_election.cc:302] T 44650e4deef944c2a35f672a8b0b824e P e3d6e62aa5b546dbaeadd8ba934aa24c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 148385836b554faaba8ee69765e73730, e3d6e62aa5b546dbaeadd8ba934aa24c; no voters:
I20260927 14:54:07.924479 14625 raft_consensus.cc:2781] T 44650e4deef944c2a35f672a8b0b824e P e3d6e62aa5b546dbaeadd8ba934aa24c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 14:54:07.924567 14625 raft_consensus.cc:482] T 44650e4deef944c2a35f672a8b0b824e P e3d6e62aa5b546dbaeadd8ba934aa24c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 14:54:07.924625 14625 raft_consensus.cc:3037] T 44650e4deef944c2a35f672a8b0b824e P e3d6e62aa5b546dbaeadd8ba934aa24c [term 0 FOLLOWER]: Advancing to term 1
I20260927 14:54:07.925680 14625 raft_consensus.cc:504] T 44650e4deef944c2a35f672a8b0b824e P e3d6e62aa5b546dbaeadd8ba934aa24c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20705812c28647ec9ef08248d5de0736" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42807 } } peers { permanent_uuid: "e3d6e62aa5b546dbaeadd8ba934aa24c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36587 } } peers { permanent_uuid: "148385836b554faaba8ee69765e73730" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32885 } }
I20260927 14:54:07.926031 14625 leader_election.cc:288] T 44650e4deef944c2a35f672a8b0b824e P e3d6e62aa5b546dbaeadd8ba934aa24c [CANDIDATE]: Term 1 election: Requested vote from peers 20705812c28647ec9ef08248d5de0736 (127.0.98.3:42807), 148385836b554faaba8ee69765e73730 (127.0.98.2:32885)
I20260927 14:54:07.926326 14555 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44650e4deef944c2a35f672a8b0b824e" candidate_uuid: "e3d6e62aa5b546dbaeadd8ba934aa24c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "20705812c28647ec9ef08248d5de0736"
I20260927 14:54:07.926450 14555 raft_consensus.cc:3037] T 44650e4deef944c2a35f672a8b0b824e P 20705812c28647ec9ef08248d5de0736 [term 0 FOLLOWER]: Advancing to term 1
I20260927 14:54:07.926462 14421 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44650e4deef944c2a35f672a8b0b824e" candidate_uuid: "e3d6e62aa5b546dbaeadd8ba934aa24c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "148385836b554faaba8ee69765e73730"
I20260927 14:54:07.926571 14421 raft_consensus.cc:3037] T 44650e4deef944c2a35f672a8b0b824e P 148385836b554faaba8ee69765e73730 [term 0 FOLLOWER]: Advancing to term 1
I20260927 14:54:07.927785 14421 raft_consensus.cc:2454] T 44650e4deef944c2a35f672a8b0b824e P 148385836b554faaba8ee69765e73730 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e3d6e62aa5b546dbaeadd8ba934aa24c in term 1.
I20260927 14:54:07.927785 14555 raft_consensus.cc:2454] T 44650e4deef944c2a35f672a8b0b824e P 20705812c28647ec9ef08248d5de0736 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e3d6e62aa5b546dbaeadd8ba934aa24c in term 1.
I20260927 14:54:07.928045 14220 leader_election.cc:302] T 44650e4deef944c2a35f672a8b0b824e P e3d6e62aa5b546dbaeadd8ba934aa24c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 148385836b554faaba8ee69765e73730, e3d6e62aa5b546dbaeadd8ba934aa24c; no voters:
I20260927 14:54:07.928233 14625 raft_consensus.cc:2781] T 44650e4deef944c2a35f672a8b0b824e P e3d6e62aa5b546dbaeadd8ba934aa24c [term 1 FOLLOWER]: Leader election won for term 1
I20260927 14:54:07.928421 14625 raft_consensus.cc:686] T 44650e4deef944c2a35f672a8b0b824e P e3d6e62aa5b546dbaeadd8ba934aa24c [term 1 LEADER]: Becoming Leader. State: Replica: e3d6e62aa5b546dbaeadd8ba934aa24c, State: Running, Role: LEADER
I20260927 14:54:07.928757 14625 consensus_queue.cc:237] T 44650e4deef944c2a35f672a8b0b824e P e3d6e62aa5b546dbaeadd8ba934aa24c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20705812c28647ec9ef08248d5de0736" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42807 } } peers { permanent_uuid: "e3d6e62aa5b546dbaeadd8ba934aa24c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36587 } } peers { permanent_uuid: "148385836b554faaba8ee69765e73730" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32885 } }
I20260927 14:54:07.931870 14140 catalog_manager.cc:5495] T 44650e4deef944c2a35f672a8b0b824e P e3d6e62aa5b546dbaeadd8ba934aa24c reported cstate change: term changed from 0 to 1, leader changed from <none> to e3d6e62aa5b546dbaeadd8ba934aa24c (127.0.98.1). New cstate: current_term: 1 leader_uuid: "e3d6e62aa5b546dbaeadd8ba934aa24c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20705812c28647ec9ef08248d5de0736" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42807 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e3d6e62aa5b546dbaeadd8ba934aa24c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36587 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "148385836b554faaba8ee69765e73730" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32885 } health_report { overall_health: UNKNOWN } } }
I20260927 14:54:07.946053 14615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:54:07.942712 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59798 (local address 127.0.98.1:36587)
0927 14:54:07.942841 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0927 14:54:07.942844 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 14:54:07.942923 (+ 79us) server_negotiation.cc:407] Connection header received
0927 14:54:07.943050 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:54:07.943052 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:54:07.943101 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:54:07.943182 (+ 81us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 14:54:07.943387 (+ 205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:54:07.943866 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:54:07.944727 (+ 861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:54:07.944922 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:54:07.944991 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:54:07.945367 (+ 376us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 14:54:07.945750 (+ 383us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 14:54:07.945838 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0927 14:54:07.945914 (+ 76us) server_negotiation.cc:299] Negotiation successful
0927 14:54:07.945956 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260927 14:54:07.951728 14421 raft_consensus.cc:1261] T 44650e4deef944c2a35f672a8b0b824e P 148385836b554faaba8ee69765e73730 [term 1 FOLLOWER]: Refusing update from remote peer e3d6e62aa5b546dbaeadd8ba934aa24c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 14:54:07.951733 14555 raft_consensus.cc:1261] T 44650e4deef944c2a35f672a8b0b824e P 20705812c28647ec9ef08248d5de0736 [term 1 FOLLOWER]: Refusing update from remote peer e3d6e62aa5b546dbaeadd8ba934aa24c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 14:54:07.952420 14629 consensus_queue.cc:1035] T 44650e4deef944c2a35f672a8b0b824e P e3d6e62aa5b546dbaeadd8ba934aa24c [LEADER]: Connected to new peer: Peer: permanent_uuid: "20705812c28647ec9ef08248d5de0736" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42807 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 14:54:07.952607 14625 consensus_queue.cc:1035] T 44650e4deef944c2a35f672a8b0b824e P e3d6e62aa5b546dbaeadd8ba934aa24c [LEADER]: Connected to new peer: Peer: permanent_uuid: "148385836b554faaba8ee69765e73730" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32885 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 14:54:07.959530 14632 mvcc.cc:205] Tried to move back new op lower bound from 7333973393203081216 to 7333973393119162368. Current Snapshot: MvccSnapshot[applied={T|T < 7333973393203081216}]
I20260927 14:54:07.960318 14633 mvcc.cc:205] Tried to move back new op lower bound from 7333973393203081216 to 7333973393119162368. Current Snapshot: MvccSnapshot[applied={T|T < 7333973393203081216}]
I20260927 14:54:07.961651 14634 mvcc.cc:205] Tried to move back new op lower bound from 7333973393203081216 to 7333973393119162368. Current Snapshot: MvccSnapshot[applied={T|T < 7333973393203081216}]
I20260927 14:54:07.967087 14139 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56838:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 14:54:07.967257 14139 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56838:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 14:54:07.969594 14139 catalog_manager.cc:5782] T 00000000000000000000000000000000 P bbbbbbf5e7484aeea9277246d74d663b: Sending DeleteTablet for 3 replicas of tablet 44650e4deef944c2a35f672a8b0b824e
I20260927 14:54:07.970317 14401 tablet_service.cc:1514] Processing DeleteTablet for tablet 44650e4deef944c2a35f672a8b0b824e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 14:54:07 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:38686
I20260927 14:54:07.970317 14535 tablet_service.cc:1514] Processing DeleteTablet for tablet 44650e4deef944c2a35f672a8b0b824e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 14:54:07 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:58538
I20260927 14:54:07.970816 14642 tablet_replica.cc:330] T 44650e4deef944c2a35f672a8b0b824e P 148385836b554faaba8ee69765e73730: stopping tablet replica
I20260927 14:54:07.970800 14267 tablet_service.cc:1514] Processing DeleteTablet for tablet 44650e4deef944c2a35f672a8b0b824e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 14:54:07 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:59788
I20260927 14:54:07.971124 14642 raft_consensus.cc:2229] T 44650e4deef944c2a35f672a8b0b824e P 148385836b554faaba8ee69765e73730 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 14:54:07.971280 14643 tablet_replica.cc:330] T 44650e4deef944c2a35f672a8b0b824e P e3d6e62aa5b546dbaeadd8ba934aa24c: stopping tablet replica
I20260927 14:54:07.971431 14642 raft_consensus.cc:2258] T 44650e4deef944c2a35f672a8b0b824e P 148385836b554faaba8ee69765e73730 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 14:54:07.971578 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6936m5/build/debug/bin/kudu with pid 14204
I20260927 14:54:07.971688 14643 raft_consensus.cc:2229] T 44650e4deef944c2a35f672a8b0b824e P e3d6e62aa5b546dbaeadd8ba934aa24c [term 1 LEADER]: Raft consensus shutting down.
I20260927 14:54:07.972334 14641 tablet_replica.cc:330] T 44650e4deef944c2a35f672a8b0b824e P 20705812c28647ec9ef08248d5de0736: stopping tablet replica
I20260927 14:54:07.972596 14641 raft_consensus.cc:2229] T 44650e4deef944c2a35f672a8b0b824e P 20705812c28647ec9ef08248d5de0736 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 14:54:07.972849 14641 raft_consensus.cc:2258] T 44650e4deef944c2a35f672a8b0b824e P 20705812c28647ec9ef08248d5de0736 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 14:54:07.972922 14642 ts_tablet_manager.cc:1905] T 44650e4deef944c2a35f672a8b0b824e P 148385836b554faaba8ee69765e73730: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 14:54:07.976394 14642 ts_tablet_manager.cc:1918] T 44650e4deef944c2a35f672a8b0b824e P 148385836b554faaba8ee69765e73730: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 14:54:07.976496 14642 log.cc:1196] T 44650e4deef944c2a35f672a8b0b824e P 148385836b554faaba8ee69765e73730: Deleting WAL directory at /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790520684602251-392-0/minicluster-data/ts-1/wal/wals/44650e4deef944c2a35f672a8b0b824e
I20260927 14:54:07.977237 14642 ts_tablet_manager.cc:1939] T 44650e4deef944c2a35f672a8b0b824e P 148385836b554faaba8ee69765e73730: Deleting consensus metadata
I20260927 14:54:07.977571 14641 ts_tablet_manager.cc:1905] T 44650e4deef944c2a35f672a8b0b824e P 20705812c28647ec9ef08248d5de0736: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 14:54:07.978438 14126 catalog_manager.cc:4841] TS 148385836b554faaba8ee69765e73730 (127.0.98.2:32885): tablet 44650e4deef944c2a35f672a8b0b824e (table test-table [id=58d23e136bf84ea09cfd849e6e3496c6]) successfully deleted
W20260927 14:54:07.979530 14129 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:36587 (error 108)
I20260927 14:54:07.979770 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6936m5/build/debug/bin/kudu with pid 14338
I20260927 14:54:07.980846 14612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:54:07.980524 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:36587 (local address 127.0.0.1:59812)
0927 14:54:07.980673 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0927 14:54:07.980750 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:36587: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260927 14:54:07.981010 14129 catalog_manager.cc:4584] TS e3d6e62aa5b546dbaeadd8ba934aa24c (127.0.98.1:36587): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 44650e4deef944c2a35f672a8b0b824e: Network error: Client connection negotiation failed: client connection to 127.0.98.1:36587: connect: Connection refused (error 111)
I20260927 14:54:07.985710 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6936m5/build/debug/bin/kudu with pid 14472
I20260927 14:54:07.991233 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6936m5/build/debug/bin/kudu with pid 14110
I20260927 14:54:07.997758 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6936m5/build/debug/bin/kudu
/tmp/dist-test-task6936m5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790520684602251-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790520684602251-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790520684602251-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790520684602251-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40583
--webserver_interface=127.0.98.62
--webserver_port=42835
--keytab_file=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790520684602251-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45035
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:40583
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790520684602251-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790520684602251-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790520684602251-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 14:54:08.105208 14645 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 14:54:08.105484 14645 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 14:54:08.105525 14645 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 14:54:08.109476 14645 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 14:54:08.109526 14645 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 14:54:08.109546 14645 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 14:54:08.109563 14645 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 14:54:08.109578 14645 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 14:54:08.114691 14645 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45035
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790520684602251-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790520684602251-392-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:40583
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790520684602251-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40583
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790520684602251-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=42835
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14645
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790520684602251-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:15:17 UTC on ccycloud.quasar-ledxfe.root.comops.site
build id 4676134
I20260927 14:54:08.115664 14645 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 14:54:08.116730 14645 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 14:54:08.129878 14652 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 14:54:08.129864 14654 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 14:54:08.129909 14651 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 14:54:08.130138 14645 server_base.cc:1034] running on GCE node
I20260927 14:54:08.130793 14645 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 14:54:08.132019 14645 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 14:54:08.133220 14645 hybrid_clock.cc:648] HybridClock initialized: now 1790520848133178 us; error 56 us; skew 500 ppm
Sep 27 14:54:08 dist-test-slave-xk1g krb5kdc[14097](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790520848, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 14:54:08.136042 14645 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 14:54:08.137030 14645 webserver.cc:460] Webserver started at http://127.0.98.62:42835/ using document root <none> and password file <none>
I20260927 14:54:08.137689 14645 fs_manager.cc:359] Metadata directory not provided
I20260927 14:54:08.137805 14645 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 14:54:08.141800 14645 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 14:54:08.144071 14661 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 14:54:08.145392 14645 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260927 14:54:08.145535 14645 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790520684602251-392-0/minicluster-data/master-0/data,/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790520684602251-392-0/minicluster-data/master-0/wal
uuid: "bbbbbbf5e7484aeea9277246d74d663b"
format_stamp: "Formatted at 2026-09-27 14:54:07 on dist-test-slave-xk1g"
server_key: "9cfcd137779d48b5a67105aae8a2a984"
server_key_iv: "6a588433d7e816f740f0b7dd7afcfe18"
server_key_version: "encryptionkey@0"
I20260927 14:54:08.146155 14645 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790520684602251-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790520684602251-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790520684602251-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 14:54:08.166754 14645 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 14:54:08.170193 14645 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 14:54:08.170408 14645 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 14:54:08.177673 14645 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:40583
I20260927 14:54:08.177973 14713 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:40583 every 8 connection(s)
I20260927 14:54:08.178843 14645 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790520684602251-392-0/minicluster-data/master-0/data/info.pb
I20260927 14:54:08.183245 14714 sys_catalog.cc:263] Verifying existing consensus state
I20260927 14:54:08.183928 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6936m5/build/debug/bin/kudu as pid 14645
I20260927 14:54:08.184415 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6936m5/build/debug/bin/kudu
/tmp/dist-test-task6936m5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790520684602251-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790520684602251-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790520684602251-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790520684602251-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:36587
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:40583
--webserver_port=35933
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790520684602251-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45035
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790520684602251-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790520684602251-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790520684602251-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 14:54:08.185565 14714 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bbbbbbf5e7484aeea9277246d74d663b: Bootstrap starting.
I20260927 14:54:08.209338 14714 log.cc:824] T 00000000000000000000000000000000 P bbbbbbf5e7484aeea9277246d74d663b: Log is configured to *not* fsync() on all Append() calls
I20260927 14:54:08.220346 14714 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bbbbbbf5e7484aeea9277246d74d663b: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 14:54:08.220804 14714 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bbbbbbf5e7484aeea9277246d74d663b: Bootstrap complete.
I20260927 14:54:08.226225 14714 raft_consensus.cc:348] T 00000000000000000000000000000000 P bbbbbbf5e7484aeea9277246d74d663b [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bbbbbbf5e7484aeea9277246d74d663b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40583 } }
I20260927 14:54:08.226737 14714 raft_consensus.cc:729] T 00000000000000000000000000000000 P bbbbbbf5e7484aeea9277246d74d663b [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: bbbbbbf5e7484aeea9277246d74d663b, State: Initialized, Role: FOLLOWER
I20260927 14:54:08.227290 14714 consensus_queue.cc:260] T 00000000000000000000000000000000 P bbbbbbf5e7484aeea9277246d74d663b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bbbbbbf5e7484aeea9277246d74d663b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40583 } }
I20260927 14:54:08.227452 14714 raft_consensus.cc:388] T 00000000000000000000000000000000 P bbbbbbf5e7484aeea9277246d74d663b [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 14:54:08.227533 14714 raft_consensus.cc:482] T 00000000000000000000000000000000 P bbbbbbf5e7484aeea9277246d74d663b [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 14:54:08.227630 14714 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bbbbbbf5e7484aeea9277246d74d663b [term 1 FOLLOWER]: Advancing to term 2
I20260927 14:54:08.230207 14714 raft_consensus.cc:504] T 00000000000000000000000000000000 P bbbbbbf5e7484aeea9277246d74d663b [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bbbbbbf5e7484aeea9277246d74d663b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40583 } }
I20260927 14:54:08.230581 14714 leader_election.cc:302] T 00000000000000000000000000000000 P bbbbbbf5e7484aeea9277246d74d663b [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bbbbbbf5e7484aeea9277246d74d663b; no voters:
I20260927 14:54:08.230937 14714 leader_election.cc:288] T 00000000000000000000000000000000 P bbbbbbf5e7484aeea9277246d74d663b [CANDIDATE]: Term 2 election: Requested vote from peers
I20260927 14:54:08.231094 14718 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bbbbbbf5e7484aeea9277246d74d663b [term 2 FOLLOWER]: Leader election won for term 2
I20260927 14:54:08.231375 14718 raft_consensus.cc:686] T 00000000000000000000000000000000 P bbbbbbf5e7484aeea9277246d74d663b [term 2 LEADER]: Becoming Leader. State: Replica: bbbbbbf5e7484aeea9277246d74d663b, State: Running, Role: LEADER
I20260927 14:54:08.231693 14718 consensus_queue.cc:237] T 00000000000000000000000000000000 P bbbbbbf5e7484aeea9277246d74d663b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bbbbbbf5e7484aeea9277246d74d663b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40583 } }
I20260927 14:54:08.232225 14714 sys_catalog.cc:564] T 00000000000000000000000000000000 P bbbbbbf5e7484aeea9277246d74d663b [sys.catalog]: configured and running, proceeding with master startup.
I20260927 14:54:08.233165 14719 sys_catalog.cc:455] T 00000000000000000000000000000000 P bbbbbbf5e7484aeea9277246d74d663b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "bbbbbbf5e7484aeea9277246d74d663b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bbbbbbf5e7484aeea9277246d74d663b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40583 } } }
I20260927 14:54:08.233290 14719 sys_catalog.cc:458] T 00000000000000000000000000000000 P bbbbbbf5e7484aeea9277246d74d663b [sys.catalog]: This master's current role is: LEADER
I20260927 14:54:08.233657 14727 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 14:54:08.233657 14720 sys_catalog.cc:455] T 00000000000000000000000000000000 P bbbbbbf5e7484aeea9277246d74d663b [sys.catalog]: SysCatalogTable state changed. Reason: New leader bbbbbbf5e7484aeea9277246d74d663b. Latest consensus state: current_term: 2 leader_uuid: "bbbbbbf5e7484aeea9277246d74d663b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bbbbbbf5e7484aeea9277246d74d663b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40583 } } }
I20260927 14:54:08.233791 14720 sys_catalog.cc:458] T 00000000000000000000000000000000 P bbbbbbf5e7484aeea9277246d74d663b [sys.catalog]: This master's current role is: LEADER
I20260927 14:54:08.238467 14727 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 14:54:08.239115 14727 catalog_manager.cc:1259] Loaded cluster ID: 8c868f16380a4788ae11a60bca2797c4
I20260927 14:54:08.239274 14727 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 14:54:08.240679 14727 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260927 14:54:08.245617 14727 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260927 14:54:08.251170 14727 catalog_manager.cc:1504] Loading token signing keys...
I20260927 14:54:08.252002 14727 catalog_manager.cc:5879] T 00000000000000000000000000000000 P bbbbbbf5e7484aeea9277246d74d663b: Loaded TSK: 0
I20260927 14:54:08.253414 14727 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260927 14:54:08.316746 14716 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 14:54:08.317243 14716 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 14:54:08.317356 14716 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 14:54:08.321633 14716 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 14:54:08.321698 14716 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 14:54:08.321761 14716 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 14:54:08.327437 14716 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45035
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790520684602251-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790520684602251-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790520684602251-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:36587
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790520684602251-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=35933
--tserver_master_addrs=127.0.98.62:40583
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14716
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790520684602251-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:15:17 UTC on ccycloud.quasar-ledxfe.root.comops.site
build id 4676134
I20260927 14:54:08.328487 14716 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 14:54:08.329560 14716 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 14:54:08.339553 14742 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 14:54:08.339708 14744 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 14:54:08.339545 14741 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 14:54:08.339856 14716 server_base.cc:1034] running on GCE node
I20260927 14:54:08.340313 14716 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 14:54:08.340983 14716 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 14:54:08.342195 14716 hybrid_clock.cc:648] HybridClock initialized: now 1790520848342164 us; error 57 us; skew 500 ppm
Sep 27 14:54:08 dist-test-slave-xk1g krb5kdc[14097](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790520848, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 14:54:08.348905 14716 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 14:54:08.350047 14716 webserver.cc:460] Webserver started at http://127.0.98.1:35933/ using document root <none> and password file <none>
I20260927 14:54:08.350744 14716 fs_manager.cc:359] Metadata directory not provided
I20260927 14:54:08.350859 14716 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 14:54:08.354909 14716 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 14:54:08.357151 14751 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 14:54:08.358554 14716 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 14:54:08.358664 14716 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790520684602251-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790520684602251-392-0/minicluster-data/ts-0/wal
uuid: "e3d6e62aa5b546dbaeadd8ba934aa24c"
format_stamp: "Formatted at 2026-09-27 14:54:07 on dist-test-slave-xk1g"
server_key: "a793d8b41e7bfe4a80c054b66217298c"
server_key_iv: "b9974b52713fa99a8ccf2c7ea4c775e1"
server_key_version: "encryptionkey@0"
I20260927 14:54:08.359270 14716 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790520684602251-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790520684602251-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790520684602251-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 14:54:08.373515 14716 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 14:54:08.376976 14716 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 14:54:08.377235 14716 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 14:54:08.377931 14716 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 14:54:08.379170 14758 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260927 14:54:08.383384 14716 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260927 14:54:08.383455 14716 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.005s user 0.001s sys 0.000s
I20260927 14:54:08.383503 14716 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260927 14:54:08.385326 14716 ts_tablet_manager.cc:610] Registered 1 tablets
I20260927 14:54:08.385388 14716 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.002s sys 0.000s
I20260927 14:54:08.385645 14758 tablet_bootstrap.cc:492] T 44650e4deef944c2a35f672a8b0b824e P e3d6e62aa5b546dbaeadd8ba934aa24c: Bootstrap starting.
I20260927 14:54:08.398839 14716 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:36587
I20260927 14:54:08.400420 14716 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790520684602251-392-0/minicluster-data/ts-0/data/info.pb
I20260927 14:54:08.401257 14865 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:36587 every 8 connection(s)
I20260927 14:54:08.402472 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6936m5/build/debug/bin/kudu as pid 14716
I20260927 14:54:08.402967 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6936m5/build/debug/bin/kudu
/tmp/dist-test-task6936m5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790520684602251-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790520684602251-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790520684602251-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790520684602251-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:32885
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:40583
--webserver_port=45923
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790520684602251-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45035
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790520684602251-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790520684602251-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790520684602251-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 14:54:08.410241 14758 log.cc:824] T 44650e4deef944c2a35f672a8b0b824e P e3d6e62aa5b546dbaeadd8ba934aa24c: Log is configured to *not* fsync() on all Append() calls
Sep 27 14:54:08 dist-test-slave-xk1g krb5kdc[14097](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790520848, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 14:54:08.419588 14758 tablet_bootstrap.cc:492] T 44650e4deef944c2a35f672a8b0b824e P e3d6e62aa5b546dbaeadd8ba934aa24c: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 14:54:08.420032 14758 tablet_bootstrap.cc:492] T 44650e4deef944c2a35f672a8b0b824e P e3d6e62aa5b546dbaeadd8ba934aa24c: Bootstrap complete.
I20260927 14:54:08.421578 14758 ts_tablet_manager.cc:1397] T 44650e4deef944c2a35f672a8b0b824e P e3d6e62aa5b546dbaeadd8ba934aa24c: Time spent bootstrapping tablet: real 0.036s user 0.021s sys 0.011s
I20260927 14:54:08.424767 14758 raft_consensus.cc:348] T 44650e4deef944c2a35f672a8b0b824e P e3d6e62aa5b546dbaeadd8ba934aa24c [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20705812c28647ec9ef08248d5de0736" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42807 } } peers { permanent_uuid: "e3d6e62aa5b546dbaeadd8ba934aa24c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36587 } } peers { permanent_uuid: "148385836b554faaba8ee69765e73730" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32885 } }
I20260927 14:54:08.425318 14758 raft_consensus.cc:729] T 44650e4deef944c2a35f672a8b0b824e P e3d6e62aa5b546dbaeadd8ba934aa24c [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: e3d6e62aa5b546dbaeadd8ba934aa24c, State: Initialized, Role: FOLLOWER
I20260927 14:54:08.425820 14758 consensus_queue.cc:260] T 44650e4deef944c2a35f672a8b0b824e P e3d6e62aa5b546dbaeadd8ba934aa24c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20705812c28647ec9ef08248d5de0736" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42807 } } peers { permanent_uuid: "e3d6e62aa5b546dbaeadd8ba934aa24c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36587 } } peers { permanent_uuid: "148385836b554faaba8ee69765e73730" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32885 } }
I20260927 14:54:08.426231 14869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:54:08.403534 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40583 (local address 127.0.98.1:41193)
0927 14:54:08.405787 (+ 2253us) negotiation.cc:107] Waiting for socket to connect
0927 14:54:08.405869 (+ 82us) client_negotiation.cc:174] Beginning negotiation
0927 14:54:08.407045 (+ 1176us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:54:08.408561 (+ 1516us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:54:08.408570 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:54:08.409034 (+ 464us) client_negotiation.cc:189] Negotiated authn=SASL
0927 14:54:08.409740 (+ 706us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:54:08.409767 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:54:08.411759 (+ 1992us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:54:08.411764 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:54:08.413121 (+ 1357us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:54:08.413132 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:54:08.414343 (+ 1211us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:54:08.414347 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:54:08.414521 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:54:08.417539 (+ 3018us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 14:54:08.417560 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0927 14:54:08.419548 (+ 1988us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 14:54:08.422507 (+ 2959us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 14:54:08.422513 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 14:54:08.422526 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0927 14:54:08.422857 (+ 331us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 14:54:08.423256 (+ 399us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 14:54:08.423259 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 14:54:08.423262 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 14:54:08.423426 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 14:54:08.424204 (+ 778us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 14:54:08.424211 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 14:54:08.424483 (+ 272us) client_negotiation.cc:769] Sending connection context
0927 14:54:08.424659 (+ 176us) client_negotiation.cc:240] Negotiation successful
0927 14:54:08.425003 (+ 344us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1931,"thread_start_us":159,"threads_started":1}
I20260927 14:54:08.426847 14758 ts_tablet_manager.cc:1428] T 44650e4deef944c2a35f672a8b0b824e P e3d6e62aa5b546dbaeadd8ba934aa24c: Time spent starting tablet: real 0.005s user 0.004s sys 0.001s
I20260927 14:54:08.428359 14868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:54:08.403522 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:41193 (local address 127.0.98.62:40583)
0927 14:54:08.405465 (+ 1943us) server_negotiation.cc:206] Beginning negotiation
0927 14:54:08.405485 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0927 14:54:08.406022 (+ 537us) server_negotiation.cc:407] Connection header received
0927 14:54:08.407601 (+ 1579us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:54:08.407619 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:54:08.408100 (+ 481us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:54:08.408486 (+ 386us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:54:08.409971 (+ 1485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:54:08.411563 (+ 1592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:54:08.413265 (+ 1702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:54:08.413965 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:54:08.414137 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:54:08.419954 (+ 5817us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:54:08.419961 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:54:08.419978 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 14:54:08.420017 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:54:08.422365 (+ 2348us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:54:08.422995 (+ 630us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:54:08.423001 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:54:08.423006 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:54:08.423150 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:54:08.423531 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:54:08.423535 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:54:08.423537 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:54:08.424022 (+ 485us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:54:08.424150 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0927 14:54:08.427493 (+ 3343us) server_negotiation.cc:299] Negotiation successful
0927 14:54:08.427776 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1770,"thread_start_us":129,"threads_started":1}
I20260927 14:54:08.430996 14866 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40583
I20260927 14:54:08.431237 14866 heartbeater.cc:464] Registering TS with master...
I20260927 14:54:08.431779 14866 heartbeater.cc:511] Master 127.0.98.62:40583 requested a full tablet report, sending...
I20260927 14:54:08.434717 14678 ts_manager.cc:194] Registered new tserver with Master: e3d6e62aa5b546dbaeadd8ba934aa24c (127.0.98.1:36587)
I20260927 14:54:08.439330 14678 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:41193
I20260927 14:54:08.442082 14866 heartbeater.cc:503] Master 127.0.98.62:40583 was elected leader, sending a full tablet report...
Sep 27 14:54:08 dist-test-slave-xk1g krb5kdc[14097](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790520848, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260927 14:54:08.450737 14873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:54:08.438523 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:36587 (local address 127.0.0.1:59816)
0927 14:54:08.439636 (+ 1113us) negotiation.cc:107] Waiting for socket to connect
0927 14:54:08.439682 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0927 14:54:08.439817 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:54:08.440681 (+ 864us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:54:08.440686 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:54:08.441148 (+ 462us) client_negotiation.cc:189] Negotiated authn=SASL
0927 14:54:08.441275 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:54:08.441283 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:54:08.442820 (+ 1537us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:54:08.442823 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:54:08.443740 (+ 917us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:54:08.443767 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:54:08.444575 (+ 808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:54:08.444593 (+ 18us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:54:08.444711 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:54:08.445439 (+ 728us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 14:54:08.445454 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 14:54:08.446966 (+ 1512us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 14:54:08.448997 (+ 2031us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 14:54:08.449006 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 14:54:08.449018 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0927 14:54:08.449306 (+ 288us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 14:54:08.449604 (+ 298us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 14:54:08.449606 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 14:54:08.449607 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 14:54:08.449682 (+ 75us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 14:54:08.450155 (+ 473us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 14:54:08.450161 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 14:54:08.450304 (+ 143us) client_negotiation.cc:769] Sending connection context
0927 14:54:08.450421 (+ 117us) client_negotiation.cc:240] Negotiation successful
0927 14:54:08.450601 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":850,"thread_start_us":131,"threads_started":1}
I20260927 14:54:08.451054 14874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:54:08.438573 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59816 (local address 127.0.98.1:36587)
0927 14:54:08.439289 (+ 716us) server_negotiation.cc:206] Beginning negotiation
0927 14:54:08.439301 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 14:54:08.439950 (+ 649us) server_negotiation.cc:407] Connection header received
0927 14:54:08.440025 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:54:08.440030 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:54:08.440369 (+ 339us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:54:08.440516 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:54:08.441411 (+ 895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:54:08.442402 (+ 991us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:54:08.443900 (+ 1498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:54:08.444356 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:54:08.444456 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:54:08.447109 (+ 2653us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:54:08.447118 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:54:08.447132 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 14:54:08.447158 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:54:08.448874 (+ 1716us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:54:08.449409 (+ 535us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:54:08.449416 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:54:08.449418 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:54:08.449512 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:54:08.449761 (+ 249us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:54:08.449764 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:54:08.449766 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:54:08.450044 (+ 278us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:54:08.450168 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0927 14:54:08.450539 (+ 371us) server_negotiation.cc:299] Negotiation successful
0927 14:54:08.450709 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":617,"thread_start_us":99,"threads_started":1}
I20260927 14:54:08.452008 14799 tablet_service.cc:1514] Processing DeleteTablet for tablet 44650e4deef944c2a35f672a8b0b824e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 14:54:07 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:59816
I20260927 14:54:08.452855 14875 tablet_replica.cc:330] stopping tablet replica
I20260927 14:54:08.453205 14875 raft_consensus.cc:2229] T 44650e4deef944c2a35f672a8b0b824e P e3d6e62aa5b546dbaeadd8ba934aa24c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 14:54:08.453476 14875 raft_consensus.cc:2258] T 44650e4deef944c2a35f672a8b0b824e P e3d6e62aa5b546dbaeadd8ba934aa24c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 14:54:08.454615 14875 ts_tablet_manager.cc:1905] T 44650e4deef944c2a35f672a8b0b824e P e3d6e62aa5b546dbaeadd8ba934aa24c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 14:54:08.459079 14875 ts_tablet_manager.cc:1918] T 44650e4deef944c2a35f672a8b0b824e P e3d6e62aa5b546dbaeadd8ba934aa24c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 14:54:08.459192 14875 log.cc:1196] T 44650e4deef944c2a35f672a8b0b824e P e3d6e62aa5b546dbaeadd8ba934aa24c: Deleting WAL directory at /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790520684602251-392-0/minicluster-data/ts-0/wal/wals/44650e4deef944c2a35f672a8b0b824e
I20260927 14:54:08.459499 14875 ts_tablet_manager.cc:1939] T 44650e4deef944c2a35f672a8b0b824e P e3d6e62aa5b546dbaeadd8ba934aa24c: Deleting consensus metadata
I20260927 14:54:08.461453 14665 catalog_manager.cc:4841] TS e3d6e62aa5b546dbaeadd8ba934aa24c (127.0.98.1:36587): tablet 44650e4deef944c2a35f672a8b0b824e (table test-table [id=58d23e136bf84ea09cfd849e6e3496c6]) successfully deleted
W20260927 14:54:08.537216 14870 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 14:54:08.537441 14870 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 14:54:08.537482 14870 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 14:54:08.541369 14870 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 14:54:08.541419 14870 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 14:54:08.541474 14870 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 14:54:08.546473 14870 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45035
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790520684602251-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790520684602251-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790520684602251-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:32885
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790520684602251-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=45923
--tserver_master_addrs=127.0.98.62:40583
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14870
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790520684602251-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:15:17 UTC on ccycloud.quasar-ledxfe.root.comops.site
build id 4676134
I20260927 14:54:08.547353 14870 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 14:54:08.548251 14870 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 14:54:08.557403 14880 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 14:54:08.557435 14881 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 14:54:08.557413 14883 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 14:54:08.557539 14870 server_base.cc:1034] running on GCE node
I20260927 14:54:08.558056 14870 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 14:54:08.558653 14870 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 14:54:08.559811 14870 hybrid_clock.cc:648] HybridClock initialized: now 1790520848559797 us; error 24 us; skew 500 ppm
Sep 27 14:54:08 dist-test-slave-xk1g krb5kdc[14097](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790520848, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 14:54:08.562489 14870 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 14:54:08.563361 14870 webserver.cc:460] Webserver started at http://127.0.98.2:45923/ using document root <none> and password file <none>
I20260927 14:54:08.563920 14870 fs_manager.cc:359] Metadata directory not provided
I20260927 14:54:08.564015 14870 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 14:54:08.567737 14870 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.001s sys 0.000s
I20260927 14:54:08.569754 14890 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 14:54:08.570930 14870 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260927 14:54:08.571049 14870 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790520684602251-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790520684602251-392-0/minicluster-data/ts-1/wal
uuid: "148385836b554faaba8ee69765e73730"
format_stamp: "Formatted at 2026-09-27 14:54:07 on dist-test-slave-xk1g"
server_key: "bb1f0ed822f3151730bc37dd3e4db41f"
server_key_iv: "5b105ad590e22475019065d8d3694582"
server_key_version: "encryptionkey@0"
I20260927 14:54:08.571547 14870 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790520684602251-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790520684602251-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790520684602251-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 14:54:08.588617 14870 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 14:54:08.591248 14870 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 14:54:08.591457 14870 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 14:54:08.592022 14870 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 14:54:08.592790 14870 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 14:54:08.592834 14870 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 14:54:08.592902 14870 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 14:54:08.592933 14870 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 14:54:08.601532 14870 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:32885
I20260927 14:54:08.601562 15003 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:32885 every 8 connection(s)
I20260927 14:54:08.602458 14870 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790520684602251-392-0/minicluster-data/ts-1/data/info.pb
I20260927 14:54:08.605386 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6936m5/build/debug/bin/kudu as pid 14870
I20260927 14:54:08.605847 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6936m5/build/debug/bin/kudu
/tmp/dist-test-task6936m5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790520684602251-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790520684602251-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790520684602251-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790520684602251-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:42807
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:40583
--webserver_port=46651
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790520684602251-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45035
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790520684602251-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790520684602251-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790520684602251-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 14:54:08 dist-test-slave-xk1g krb5kdc[14097](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790520848, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 14:54:08.617761 14868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:54:08.604098 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:60267 (local address 127.0.98.62:40583)
0927 14:54:08.604227 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0927 14:54:08.604230 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 14:54:08.604944 (+ 714us) server_negotiation.cc:407] Connection header received
0927 14:54:08.606631 (+ 1687us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:54:08.606634 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:54:08.606683 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:54:08.606754 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:54:08.608122 (+ 1368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:54:08.608989 (+ 867us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:54:08.610178 (+ 1189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:54:08.610651 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:54:08.610778 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:54:08.613467 (+ 2689us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:54:08.613474 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:54:08.613497 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 14:54:08.613538 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:54:08.615369 (+ 1831us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:54:08.615855 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:54:08.615859 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:54:08.615861 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:54:08.615965 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:54:08.616326 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:54:08.616332 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:54:08.616337 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:54:08.616615 (+ 278us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:54:08.616743 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0927 14:54:08.617345 (+ 602us) server_negotiation.cc:299] Negotiation successful
0927 14:54:08.617509 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260927 14:54:08.618300 15006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:54:08.604382 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40583 (local address 127.0.98.2:60267)
0927 14:54:08.604796 (+ 414us) negotiation.cc:107] Waiting for socket to connect
0927 14:54:08.604835 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0927 14:54:08.606414 (+ 1579us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:54:08.606922 (+ 508us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:54:08.606933 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:54:08.607448 (+ 515us) client_negotiation.cc:189] Negotiated authn=SASL
0927 14:54:08.607974 (+ 526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:54:08.607987 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:54:08.609177 (+ 1190us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:54:08.609180 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:54:08.610052 (+ 872us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:54:08.610062 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:54:08.610807 (+ 745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:54:08.610810 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:54:08.610945 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:54:08.611526 (+ 581us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 14:54:08.611541 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 14:54:08.613292 (+ 1751us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 14:54:08.615496 (+ 2204us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 14:54:08.615501 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 14:54:08.615511 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0927 14:54:08.615750 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 14:54:08.616100 (+ 350us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 14:54:08.616103 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 14:54:08.616104 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 14:54:08.616224 (+ 120us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 14:54:08.616725 (+ 501us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 14:54:08.616730 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 14:54:08.617027 (+ 297us) client_negotiation.cc:769] Sending connection context
0927 14:54:08.617250 (+ 223us) client_negotiation.cc:240] Negotiation successful
0927 14:54:08.617484 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":242,"thread_start_us":121,"threads_started":1}
I20260927 14:54:08.619432 15004 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40583
I20260927 14:54:08.619642 15004 heartbeater.cc:464] Registering TS with master...
I20260927 14:54:08.620074 15004 heartbeater.cc:511] Master 127.0.98.62:40583 requested a full tablet report, sending...
I20260927 14:54:08.621120 14678 ts_manager.cc:194] Registered new tserver with Master: 148385836b554faaba8ee69765e73730 (127.0.98.2:32885)
I20260927 14:54:08.621657 14678 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:60267
W20260927 14:54:08.718123 15007 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 14:54:08.718364 15007 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 14:54:08.718406 15007 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 14:54:08.722926 15007 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 14:54:08.722981 15007 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 14:54:08.723084 15007 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 14:54:08.728147 15007 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45035
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790520684602251-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790520684602251-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790520684602251-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:42807
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790520684602251-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=46651
--tserver_master_addrs=127.0.98.62:40583
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15007
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790520684602251-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:15:17 UTC on ccycloud.quasar-ledxfe.root.comops.site
build id 4676134
I20260927 14:54:08.729148 15007 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 14:54:08.730154 15007 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 14:54:08.739820 15013 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 14:54:08.740020 15007 server_base.cc:1034] running on GCE node
W20260927 14:54:08.739820 15012 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 14:54:08.739830 15015 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 14:54:08.740665 15007 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 14:54:08.741353 15007 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 14:54:08.742545 15007 hybrid_clock.cc:648] HybridClock initialized: now 1790520848742509 us; error 45 us; skew 500 ppm
Sep 27 14:54:08 dist-test-slave-xk1g krb5kdc[14097](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790520848, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 14:54:08.745255 15007 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 14:54:08.746093 15007 webserver.cc:460] Webserver started at http://127.0.98.3:46651/ using document root <none> and password file <none>
I20260927 14:54:08.746598 15007 fs_manager.cc:359] Metadata directory not provided
I20260927 14:54:08.746665 15007 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 14:54:08.750689 15007 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260927 14:54:08.752832 15022 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 14:54:08.754267 15007 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 14:54:08.754364 15007 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790520684602251-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790520684602251-392-0/minicluster-data/ts-2/wal
uuid: "20705812c28647ec9ef08248d5de0736"
format_stamp: "Formatted at 2026-09-27 14:54:07 on dist-test-slave-xk1g"
server_key: "b8a0672c73a73a48a6e0652b1e6efab0"
server_key_iv: "b8386891c67ea1e1632665d2d00fb3dd"
server_key_version: "encryptionkey@0"
I20260927 14:54:08.754871 15007 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790520684602251-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790520684602251-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790520684602251-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 14:54:08.765874 15007 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 14:54:08.768721 15007 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 14:54:08.768859 15007 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 14:54:08.769411 15007 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 14:54:08.770339 15029 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
W20260927 14:54:08.773898 15029 ts_tablet_manager.cc:1852] T 44650e4deef944c2a35f672a8b0b824e P 20705812c28647ec9ef08248d5de0736: Tablet Manager startup: Rolling forward tablet deletion of type TABLET_DATA_DELETED
I20260927 14:54:08.774281 15029 ts_tablet_manager.cc:1905] T 44650e4deef944c2a35f672a8b0b824e P 20705812c28647ec9ef08248d5de0736: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 14:54:08.777904 15029 ts_tablet_manager.cc:1918] T 44650e4deef944c2a35f672a8b0b824e P 20705812c28647ec9ef08248d5de0736: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 14:54:08.778019 15029 log.cc:1196] T 44650e4deef944c2a35f672a8b0b824e P 20705812c28647ec9ef08248d5de0736: Deleting WAL directory at /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790520684602251-392-0/minicluster-data/ts-2/wal/wals/44650e4deef944c2a35f672a8b0b824e
I20260927 14:54:08.778363 15029 ts_tablet_manager.cc:1939] T 44650e4deef944c2a35f672a8b0b824e P 20705812c28647ec9ef08248d5de0736: Deleting consensus metadata
I20260927 14:54:08.778656 15007 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 0 live tablets)
I20260927 14:54:08.778715 15007 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.009s user 0.000s sys 0.000s
I20260927 14:54:08.778789 15007 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 14:54:08.778826 15007 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 14:54:08.787328 15007 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:42807
I20260927 14:54:08.787374 15136 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:42807 every 8 connection(s)
I20260927 14:54:08.788132 15007 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790520684602251-392-0/minicluster-data/ts-2/data/info.pb
I20260927 14:54:08.792642 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6936m5/build/debug/bin/kudu as pid 15007
Sep 27 14:54:08 dist-test-slave-xk1g krb5kdc[14097](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790520848, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 14:54:08.801421 15140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:54:08.793361 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56852 (local address 127.0.98.62:40583)
0927 14:54:08.793578 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0927 14:54:08.793582 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 14:54:08.793731 (+ 149us) server_negotiation.cc:407] Connection header received
0927 14:54:08.793870 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:54:08.793872 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:54:08.793909 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:54:08.793976 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:54:08.794841 (+ 865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:54:08.795417 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:54:08.796521 (+ 1104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:54:08.796736 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:54:08.796812 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:54:08.798245 (+ 1433us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:54:08.798252 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:54:08.798257 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 14:54:08.798285 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:54:08.799663 (+ 1378us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:54:08.800140 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:54:08.800145 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:54:08.800146 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:54:08.800254 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:54:08.800511 (+ 257us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:54:08.800514 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:54:08.800517 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:54:08.800742 (+ 225us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:54:08.800938 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0927 14:54:08.801111 (+ 173us) server_negotiation.cc:299] Negotiation successful
0927 14:54:08.801228 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":152,"thread_start_us":71,"threads_started":1}
I20260927 14:54:08.802270 14868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:54:08.789851 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:48509 (local address 127.0.98.62:40583)
0927 14:54:08.790020 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0927 14:54:08.790024 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 14:54:08.790811 (+ 787us) server_negotiation.cc:407] Connection header received
0927 14:54:08.791615 (+ 804us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:54:08.791619 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:54:08.791667 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:54:08.791741 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:54:08.792976 (+ 1235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:54:08.793692 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:54:08.795217 (+ 1525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:54:08.795549 (+ 332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:54:08.795631 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:54:08.798341 (+ 2710us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:54:08.798344 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:54:08.798345 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 14:54:08.798376 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:54:08.799983 (+ 1607us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:54:08.800595 (+ 612us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:54:08.800598 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:54:08.800599 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:54:08.800666 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:54:08.801044 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:54:08.801047 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:54:08.801049 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:54:08.801293 (+ 244us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:54:08.801390 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0927 14:54:08.802004 (+ 614us) server_negotiation.cc:299] Negotiation successful
0927 14:54:08.802110 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260927 14:54:08.803109 15139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:54:08.790240 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40583 (local address 127.0.98.3:48509)
0927 14:54:08.790657 (+ 417us) negotiation.cc:107] Waiting for socket to connect
0927 14:54:08.790696 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0927 14:54:08.791438 (+ 742us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:54:08.791892 (+ 454us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:54:08.791899 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:54:08.792208 (+ 309us) client_negotiation.cc:189] Negotiated authn=SASL
0927 14:54:08.792834 (+ 626us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:54:08.792854 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:54:08.793846 (+ 992us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:54:08.793851 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:54:08.795112 (+ 1261us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:54:08.795120 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:54:08.795690 (+ 570us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:54:08.795695 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:54:08.795888 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:54:08.796482 (+ 594us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 14:54:08.796505 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0927 14:54:08.798179 (+ 1674us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 14:54:08.800120 (+ 1941us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 14:54:08.800125 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 14:54:08.800139 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0927 14:54:08.800503 (+ 364us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 14:54:08.800755 (+ 252us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 14:54:08.800759 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 14:54:08.800763 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 14:54:08.800937 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 14:54:08.801436 (+ 499us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 14:54:08.801441 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 14:54:08.801759 (+ 318us) client_negotiation.cc:769] Sending connection context
0927 14:54:08.801953 (+ 194us) client_negotiation.cc:240] Negotiation successful
0927 14:54:08.802266 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":235,"thread_start_us":100,"threads_started":1}
I20260927 14:54:08.804255 15137 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40583
I20260927 14:54:08.804438 15137 heartbeater.cc:464] Registering TS with master...
I20260927 14:54:08.804831 15137 heartbeater.cc:511] Master 127.0.98.62:40583 requested a full tablet report, sending...
I20260927 14:54:08.805814 14678 ts_manager.cc:194] Registered new tserver with Master: 20705812c28647ec9ef08248d5de0736 (127.0.98.3:42807)
I20260927 14:54:08.806349 14678 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:48509
I20260927 14:54:08.814749 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 14:54:08.818701 14868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:54:08.815319 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56868 (local address 127.0.98.62:40583)
0927 14:54:08.815435 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0927 14:54:08.815439 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 14:54:08.815620 (+ 181us) server_negotiation.cc:407] Connection header received
0927 14:54:08.815753 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:54:08.815755 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:54:08.815842 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:54:08.815926 (+ 84us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 14:54:08.816122 (+ 196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:54:08.816604 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:54:08.817557 (+ 953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:54:08.817760 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:54:08.817835 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:54:08.818078 (+ 243us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 14:54:08.818431 (+ 353us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 14:54:08.818515 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0927 14:54:08.818572 (+ 57us) server_negotiation.cc:299] Negotiation successful
0927 14:54:08.818613 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260927 14:54:08.819281 14678 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:56868:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 14:54:08.822516 14678 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 14:54:08.829841 14799 tablet_service.cc:1467] Processing CreateTablet for tablet 44b03fc9151b4b4ab88f7d1c5d86d2a0 (DEFAULT_TABLE table=test-table [id=2feaa6cc2b444560bcdbffd60d0956f6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 14:54:08.830494 14799 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 44b03fc9151b4b4ab88f7d1c5d86d2a0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 14:54:08.833395 14758 tablet_bootstrap.cc:492] T 44b03fc9151b4b4ab88f7d1c5d86d2a0 P e3d6e62aa5b546dbaeadd8ba934aa24c: Bootstrap starting.
I20260927 14:54:08.834357 14758 tablet_bootstrap.cc:654] T 44b03fc9151b4b4ab88f7d1c5d86d2a0 P e3d6e62aa5b546dbaeadd8ba934aa24c: Neither blocks nor log segments found. Creating new log.
I20260927 14:54:08.834419 14873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:54:08.828789 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:42807 (local address 127.0.0.1:58552)
0927 14:54:08.828916 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0927 14:54:08.828931 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 14:54:08.829011 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:54:08.830245 (+ 1234us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:54:08.830248 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:54:08.830265 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 14:54:08.830351 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:54:08.830357 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:54:08.831709 (+ 1352us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:54:08.831712 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:54:08.832679 (+ 967us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:54:08.832688 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:54:08.833901 (+ 1213us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:54:08.833904 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:54:08.834007 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:54:08.834053 (+ 46us) client_negotiation.cc:769] Sending connection context
0927 14:54:08.834178 (+ 125us) client_negotiation.cc:240] Negotiation successful
0927 14:54:08.834278 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":56}
I20260927 14:54:08.834904 15147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:54:08.829360 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:32885 (local address 127.0.0.1:38688)
0927 14:54:08.829610 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0927 14:54:08.829624 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 14:54:08.829721 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:54:08.830425 (+ 704us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:54:08.830428 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:54:08.830445 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 14:54:08.830537 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:54:08.830542 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:54:08.832041 (+ 1499us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:54:08.832046 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:54:08.833220 (+ 1174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:54:08.833230 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:54:08.834498 (+ 1268us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:54:08.834502 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:54:08.834603 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:54:08.834635 (+ 32us) client_negotiation.cc:769] Sending connection context
0927 14:54:08.834713 (+ 78us) client_negotiation.cc:240] Negotiation successful
0927 14:54:08.834775 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":110,"threads_started":1}
I20260927 14:54:08.834920 15146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:54:08.828948 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58552 (local address 127.0.98.3:42807)
0927 14:54:08.829732 (+ 784us) server_negotiation.cc:206] Beginning negotiation
0927 14:54:08.829744 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 14:54:08.829768 (+ 24us) server_negotiation.cc:407] Connection header received
0927 14:54:08.829849 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:54:08.829855 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:54:08.830073 (+ 218us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:54:08.830246 (+ 173us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 14:54:08.830488 (+ 242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:54:08.831546 (+ 1058us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:54:08.832801 (+ 1255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:54:08.833771 (+ 970us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:54:08.833894 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:54:08.834079 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0927 14:54:08.834678 (+ 599us) server_negotiation.cc:299] Negotiation successful
0927 14:54:08.834776 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":673,"thread_start_us":209,"threads_started":1}
I20260927 14:54:08.835439 15148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:54:08.829549 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38688 (local address 127.0.98.2:32885)
0927 14:54:08.829955 (+ 406us) server_negotiation.cc:206] Beginning negotiation
0927 14:54:08.829965 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 14:54:08.829985 (+ 20us) server_negotiation.cc:407] Connection header received
0927 14:54:08.830050 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:54:08.830055 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:54:08.830224 (+ 169us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:54:08.830350 (+ 126us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 14:54:08.830688 (+ 338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:54:08.831864 (+ 1176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:54:08.833361 (+ 1497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:54:08.834210 (+ 849us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:54:08.834300 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:54:08.834434 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0927 14:54:08.835212 (+ 778us) server_negotiation.cc:299] Negotiation successful
0927 14:54:08.835307 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":326,"thread_start_us":119,"threads_started":1}
I20260927 14:54:08.836664 14758 tablet_bootstrap.cc:492] T 44b03fc9151b4b4ab88f7d1c5d86d2a0 P e3d6e62aa5b546dbaeadd8ba934aa24c: No bootstrap required, opened a new log
I20260927 14:54:08.836761 14758 ts_tablet_manager.cc:1397] T 44b03fc9151b4b4ab88f7d1c5d86d2a0 P e3d6e62aa5b546dbaeadd8ba934aa24c: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.001s
I20260927 14:54:08.836381 15071 tablet_service.cc:1467] Processing CreateTablet for tablet 44b03fc9151b4b4ab88f7d1c5d86d2a0 (DEFAULT_TABLE table=test-table [id=2feaa6cc2b444560bcdbffd60d0956f6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 14:54:08.837245 14758 raft_consensus.cc:348] T 44b03fc9151b4b4ab88f7d1c5d86d2a0 P e3d6e62aa5b546dbaeadd8ba934aa24c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20705812c28647ec9ef08248d5de0736" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42807 } } peers { permanent_uuid: "148385836b554faaba8ee69765e73730" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32885 } } peers { permanent_uuid: "e3d6e62aa5b546dbaeadd8ba934aa24c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36587 } }
I20260927 14:54:08.837369 14758 raft_consensus.cc:374] T 44b03fc9151b4b4ab88f7d1c5d86d2a0 P e3d6e62aa5b546dbaeadd8ba934aa24c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 14:54:08.837393 15071 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 44b03fc9151b4b4ab88f7d1c5d86d2a0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 14:54:08.837405 14758 raft_consensus.cc:729] T 44b03fc9151b4b4ab88f7d1c5d86d2a0 P e3d6e62aa5b546dbaeadd8ba934aa24c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e3d6e62aa5b546dbaeadd8ba934aa24c, State: Initialized, Role: FOLLOWER
I20260927 14:54:08.837524 14758 consensus_queue.cc:260] T 44b03fc9151b4b4ab88f7d1c5d86d2a0 P e3d6e62aa5b546dbaeadd8ba934aa24c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20705812c28647ec9ef08248d5de0736" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42807 } } peers { permanent_uuid: "148385836b554faaba8ee69765e73730" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32885 } } peers { permanent_uuid: "e3d6e62aa5b546dbaeadd8ba934aa24c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36587 } }
I20260927 14:54:08.837769 14758 ts_tablet_manager.cc:1428] T 44b03fc9151b4b4ab88f7d1c5d86d2a0 P e3d6e62aa5b546dbaeadd8ba934aa24c: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260927 14:54:08.838470 14938 tablet_service.cc:1467] Processing CreateTablet for tablet 44b03fc9151b4b4ab88f7d1c5d86d2a0 (DEFAULT_TABLE table=test-table [id=2feaa6cc2b444560bcdbffd60d0956f6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 14:54:08.839560 14938 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 44b03fc9151b4b4ab88f7d1c5d86d2a0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 14:54:08.841845 15029 tablet_bootstrap.cc:492] T 44b03fc9151b4b4ab88f7d1c5d86d2a0 P 20705812c28647ec9ef08248d5de0736: Bootstrap starting.
I20260927 14:54:08.844007 15029 tablet_bootstrap.cc:654] T 44b03fc9151b4b4ab88f7d1c5d86d2a0 P 20705812c28647ec9ef08248d5de0736: Neither blocks nor log segments found. Creating new log.
I20260927 14:54:08.844700 15029 log.cc:824] T 44b03fc9151b4b4ab88f7d1c5d86d2a0 P 20705812c28647ec9ef08248d5de0736: Log is configured to *not* fsync() on all Append() calls
I20260927 14:54:08.846385 15029 tablet_bootstrap.cc:492] T 44b03fc9151b4b4ab88f7d1c5d86d2a0 P 20705812c28647ec9ef08248d5de0736: No bootstrap required, opened a new log
I20260927 14:54:08.846554 15029 ts_tablet_manager.cc:1397] T 44b03fc9151b4b4ab88f7d1c5d86d2a0 P 20705812c28647ec9ef08248d5de0736: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.002s
I20260927 14:54:08.846797 15151 tablet_bootstrap.cc:492] T 44b03fc9151b4b4ab88f7d1c5d86d2a0 P 148385836b554faaba8ee69765e73730: Bootstrap starting.
I20260927 14:54:08.849426 15151 tablet_bootstrap.cc:654] T 44b03fc9151b4b4ab88f7d1c5d86d2a0 P 148385836b554faaba8ee69765e73730: Neither blocks nor log segments found. Creating new log.
I20260927 14:54:08.850281 15151 log.cc:824] T 44b03fc9151b4b4ab88f7d1c5d86d2a0 P 148385836b554faaba8ee69765e73730: Log is configured to *not* fsync() on all Append() calls
I20260927 14:54:08.850744 15029 raft_consensus.cc:348] T 44b03fc9151b4b4ab88f7d1c5d86d2a0 P 20705812c28647ec9ef08248d5de0736 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20705812c28647ec9ef08248d5de0736" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42807 } } peers { permanent_uuid: "148385836b554faaba8ee69765e73730" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32885 } } peers { permanent_uuid: "e3d6e62aa5b546dbaeadd8ba934aa24c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36587 } }
I20260927 14:54:08.850986 15029 raft_consensus.cc:374] T 44b03fc9151b4b4ab88f7d1c5d86d2a0 P 20705812c28647ec9ef08248d5de0736 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 14:54:08.851080 15029 raft_consensus.cc:729] T 44b03fc9151b4b4ab88f7d1c5d86d2a0 P 20705812c28647ec9ef08248d5de0736 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 20705812c28647ec9ef08248d5de0736, State: Initialized, Role: FOLLOWER
I20260927 14:54:08.851528 15029 consensus_queue.cc:260] T 44b03fc9151b4b4ab88f7d1c5d86d2a0 P 20705812c28647ec9ef08248d5de0736 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20705812c28647ec9ef08248d5de0736" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42807 } } peers { permanent_uuid: "148385836b554faaba8ee69765e73730" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32885 } } peers { permanent_uuid: "e3d6e62aa5b546dbaeadd8ba934aa24c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36587 } }
I20260927 14:54:08.852203 15137 heartbeater.cc:503] Master 127.0.98.62:40583 was elected leader, sending a full tablet report...
I20260927 14:54:08.852468 15029 ts_tablet_manager.cc:1428] T 44b03fc9151b4b4ab88f7d1c5d86d2a0 P 20705812c28647ec9ef08248d5de0736: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260927 14:54:08.852489 15151 tablet_bootstrap.cc:492] T 44b03fc9151b4b4ab88f7d1c5d86d2a0 P 148385836b554faaba8ee69765e73730: No bootstrap required, opened a new log
I20260927 14:54:08.852614 15151 ts_tablet_manager.cc:1397] T 44b03fc9151b4b4ab88f7d1c5d86d2a0 P 148385836b554faaba8ee69765e73730: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.004s
I20260927 14:54:08.855737 15151 raft_consensus.cc:348] T 44b03fc9151b4b4ab88f7d1c5d86d2a0 P 148385836b554faaba8ee69765e73730 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20705812c28647ec9ef08248d5de0736" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42807 } } peers { permanent_uuid: "148385836b554faaba8ee69765e73730" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32885 } } peers { permanent_uuid: "e3d6e62aa5b546dbaeadd8ba934aa24c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36587 } }
I20260927 14:54:08.855937 15151 raft_consensus.cc:374] T 44b03fc9151b4b4ab88f7d1c5d86d2a0 P 148385836b554faaba8ee69765e73730 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 14:54:08.855995 15151 raft_consensus.cc:729] T 44b03fc9151b4b4ab88f7d1c5d86d2a0 P 148385836b554faaba8ee69765e73730 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 148385836b554faaba8ee69765e73730, State: Initialized, Role: FOLLOWER
I20260927 14:54:08.856422 15151 consensus_queue.cc:260] T 44b03fc9151b4b4ab88f7d1c5d86d2a0 P 148385836b554faaba8ee69765e73730 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20705812c28647ec9ef08248d5de0736" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42807 } } peers { permanent_uuid: "148385836b554faaba8ee69765e73730" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32885 } } peers { permanent_uuid: "e3d6e62aa5b546dbaeadd8ba934aa24c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36587 } }
I20260927 14:54:08.857164 15004 heartbeater.cc:503] Master 127.0.98.62:40583 was elected leader, sending a full tablet report...
I20260927 14:54:08.857462 15151 ts_tablet_manager.cc:1428] T 44b03fc9151b4b4ab88f7d1c5d86d2a0 P 148385836b554faaba8ee69765e73730: Time spent starting tablet: real 0.005s user 0.005s sys 0.001s
W20260927 14:54:08.908336 14867 tablet.cc:2369] T 44b03fc9151b4b4ab88f7d1c5d86d2a0 P e3d6e62aa5b546dbaeadd8ba934aa24c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 14:54:09.030663 14872 raft_consensus.cc:482] T 44b03fc9151b4b4ab88f7d1c5d86d2a0 P e3d6e62aa5b546dbaeadd8ba934aa24c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 14:54:09.030876 14872 raft_consensus.cc:504] T 44b03fc9151b4b4ab88f7d1c5d86d2a0 P e3d6e62aa5b546dbaeadd8ba934aa24c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20705812c28647ec9ef08248d5de0736" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42807 } } peers { permanent_uuid: "148385836b554faaba8ee69765e73730" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32885 } } peers { permanent_uuid: "e3d6e62aa5b546dbaeadd8ba934aa24c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36587 } }
I20260927 14:54:09.031945 14872 leader_election.cc:288] T 44b03fc9151b4b4ab88f7d1c5d86d2a0 P e3d6e62aa5b546dbaeadd8ba934aa24c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 20705812c28647ec9ef08248d5de0736 (127.0.98.3:42807), 148385836b554faaba8ee69765e73730 (127.0.98.2:32885)
I20260927 14:54:09.037326 15146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:54:09.032351 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:56695 (local address 127.0.98.3:42807)
0927 14:54:09.032479 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0927 14:54:09.032483 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 14:54:09.032578 (+ 95us) server_negotiation.cc:407] Connection header received
0927 14:54:09.032783 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:54:09.032786 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:54:09.032839 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:54:09.032925 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 14:54:09.033319 (+ 394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:54:09.034112 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:54:09.035930 (+ 1818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:54:09.036562 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:54:09.036675 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:54:09.036743 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0927 14:54:09.037100 (+ 357us) server_negotiation.cc:299] Negotiation successful
0927 14:54:09.037182 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260927 14:54:09.037326 15148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:54:09.032303 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:54661 (local address 127.0.98.2:32885)
0927 14:54:09.032539 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0927 14:54:09.032545 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 14:54:09.032563 (+ 18us) server_negotiation.cc:407] Connection header received
0927 14:54:09.032783 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:54:09.032786 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:54:09.032840 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:54:09.032927 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 14:54:09.033302 (+ 375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:54:09.034116 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:54:09.035930 (+ 1814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:54:09.036562 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:54:09.036675 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:54:09.036743 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0927 14:54:09.037100 (+ 357us) server_negotiation.cc:299] Negotiation successful
0927 14:54:09.037182 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":124}
I20260927 14:54:09.037392 15156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:54:09.032185 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:32885 (local address 127.0.98.1:54661)
0927 14:54:09.032488 (+ 303us) negotiation.cc:107] Waiting for socket to connect
0927 14:54:09.032503 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 14:54:09.032637 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:54:09.032957 (+ 320us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:54:09.032960 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:54:09.032985 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 14:54:09.033185 (+ 200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:54:09.033191 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:54:09.034231 (+ 1040us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:54:09.034234 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:54:09.035797 (+ 1563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:54:09.035802 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:54:09.036717 (+ 915us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:54:09.036721 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:54:09.036873 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:54:09.036959 (+ 86us) client_negotiation.cc:769] Sending connection context
0927 14:54:09.037043 (+ 84us) client_negotiation.cc:240] Negotiation successful
0927 14:54:09.037188 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":226,"mutex_wait_us":261,"spinlock_wait_cycles":17280,"thread_start_us":90,"threads_started":1}
I20260927 14:54:09.037395 15155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:54:09.032193 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:42807 (local address 127.0.98.1:56695)
0927 14:54:09.032508 (+ 315us) negotiation.cc:107] Waiting for socket to connect
0927 14:54:09.032522 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 14:54:09.032637 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:54:09.032957 (+ 320us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:54:09.032960 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:54:09.032985 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 14:54:09.033186 (+ 201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:54:09.033191 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:54:09.034231 (+ 1040us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:54:09.034234 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:54:09.035787 (+ 1553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:54:09.035800 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:54:09.036717 (+ 917us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:54:09.036721 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:54:09.036870 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:54:09.036969 (+ 99us) client_negotiation.cc:769] Sending connection context
0927 14:54:09.037043 (+ 74us) client_negotiation.cc:240] Negotiation successful
0927 14:54:09.037189 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":265,"mutex_wait_us":266,"spinlock_wait_cycles":23040,"thread_start_us":85,"threads_started":1}
I20260927 14:54:09.038174 14958 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44b03fc9151b4b4ab88f7d1c5d86d2a0" candidate_uuid: "e3d6e62aa5b546dbaeadd8ba934aa24c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "148385836b554faaba8ee69765e73730" is_pre_election: true
I20260927 14:54:09.038298 15091 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44b03fc9151b4b4ab88f7d1c5d86d2a0" candidate_uuid: "e3d6e62aa5b546dbaeadd8ba934aa24c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "20705812c28647ec9ef08248d5de0736" is_pre_election: true
I20260927 14:54:09.038542 14958 raft_consensus.cc:2454] T 44b03fc9151b4b4ab88f7d1c5d86d2a0 P 148385836b554faaba8ee69765e73730 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e3d6e62aa5b546dbaeadd8ba934aa24c in term 0.
I20260927 14:54:09.038609 15091 raft_consensus.cc:2454] T 44b03fc9151b4b4ab88f7d1c5d86d2a0 P 20705812c28647ec9ef08248d5de0736 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e3d6e62aa5b546dbaeadd8ba934aa24c in term 0.
W20260927 14:54:09.038937 15138 tablet.cc:2369] T 44b03fc9151b4b4ab88f7d1c5d86d2a0 P 20705812c28647ec9ef08248d5de0736: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 14:54:09.039036 14752 leader_election.cc:302] T 44b03fc9151b4b4ab88f7d1c5d86d2a0 P e3d6e62aa5b546dbaeadd8ba934aa24c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 148385836b554faaba8ee69765e73730, e3d6e62aa5b546dbaeadd8ba934aa24c; no voters:
I20260927 14:54:09.039359 14872 raft_consensus.cc:2781] T 44b03fc9151b4b4ab88f7d1c5d86d2a0 P e3d6e62aa5b546dbaeadd8ba934aa24c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 14:54:09.039460 14872 raft_consensus.cc:482] T 44b03fc9151b4b4ab88f7d1c5d86d2a0 P e3d6e62aa5b546dbaeadd8ba934aa24c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 14:54:09.039528 14872 raft_consensus.cc:3037] T 44b03fc9151b4b4ab88f7d1c5d86d2a0 P e3d6e62aa5b546dbaeadd8ba934aa24c [term 0 FOLLOWER]: Advancing to term 1
I20260927 14:54:09.040537 14872 raft_consensus.cc:504] T 44b03fc9151b4b4ab88f7d1c5d86d2a0 P e3d6e62aa5b546dbaeadd8ba934aa24c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20705812c28647ec9ef08248d5de0736" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42807 } } peers { permanent_uuid: "148385836b554faaba8ee69765e73730" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32885 } } peers { permanent_uuid: "e3d6e62aa5b546dbaeadd8ba934aa24c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36587 } }
I20260927 14:54:09.040918 14872 leader_election.cc:288] T 44b03fc9151b4b4ab88f7d1c5d86d2a0 P e3d6e62aa5b546dbaeadd8ba934aa24c [CANDIDATE]: Term 1 election: Requested vote from peers 20705812c28647ec9ef08248d5de0736 (127.0.98.3:42807), 148385836b554faaba8ee69765e73730 (127.0.98.2:32885)
I20260927 14:54:09.041333 15091 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44b03fc9151b4b4ab88f7d1c5d86d2a0" candidate_uuid: "e3d6e62aa5b546dbaeadd8ba934aa24c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "20705812c28647ec9ef08248d5de0736"
I20260927 14:54:09.041472 15091 raft_consensus.cc:3037] T 44b03fc9151b4b4ab88f7d1c5d86d2a0 P 20705812c28647ec9ef08248d5de0736 [term 0 FOLLOWER]: Advancing to term 1
I20260927 14:54:09.042502 15091 raft_consensus.cc:2454] T 44b03fc9151b4b4ab88f7d1c5d86d2a0 P 20705812c28647ec9ef08248d5de0736 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e3d6e62aa5b546dbaeadd8ba934aa24c in term 1.
I20260927 14:54:09.042829 14754 leader_election.cc:302] T 44b03fc9151b4b4ab88f7d1c5d86d2a0 P e3d6e62aa5b546dbaeadd8ba934aa24c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20705812c28647ec9ef08248d5de0736, e3d6e62aa5b546dbaeadd8ba934aa24c; no voters:
I20260927 14:54:09.042815 14958 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44b03fc9151b4b4ab88f7d1c5d86d2a0" candidate_uuid: "e3d6e62aa5b546dbaeadd8ba934aa24c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "148385836b554faaba8ee69765e73730"
I20260927 14:54:09.042949 14958 raft_consensus.cc:3037] T 44b03fc9151b4b4ab88f7d1c5d86d2a0 P 148385836b554faaba8ee69765e73730 [term 0 FOLLOWER]: Advancing to term 1
I20260927 14:54:09.043951 14958 raft_consensus.cc:2454] T 44b03fc9151b4b4ab88f7d1c5d86d2a0 P 148385836b554faaba8ee69765e73730 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e3d6e62aa5b546dbaeadd8ba934aa24c in term 1.
I20260927 14:54:09.044163 14872 raft_consensus.cc:2781] T 44b03fc9151b4b4ab88f7d1c5d86d2a0 P e3d6e62aa5b546dbaeadd8ba934aa24c [term 1 FOLLOWER]: Leader election won for term 1
I20260927 14:54:09.044399 14872 raft_consensus.cc:686] T 44b03fc9151b4b4ab88f7d1c5d86d2a0 P e3d6e62aa5b546dbaeadd8ba934aa24c [term 1 LEADER]: Becoming Leader. State: Replica: e3d6e62aa5b546dbaeadd8ba934aa24c, State: Running, Role: LEADER
I20260927 14:54:09.044665 14872 consensus_queue.cc:237] T 44b03fc9151b4b4ab88f7d1c5d86d2a0 P e3d6e62aa5b546dbaeadd8ba934aa24c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20705812c28647ec9ef08248d5de0736" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42807 } } peers { permanent_uuid: "148385836b554faaba8ee69765e73730" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32885 } } peers { permanent_uuid: "e3d6e62aa5b546dbaeadd8ba934aa24c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36587 } }
I20260927 14:54:09.047770 14678 catalog_manager.cc:5495] T 44b03fc9151b4b4ab88f7d1c5d86d2a0 P e3d6e62aa5b546dbaeadd8ba934aa24c reported cstate change: term changed from 0 to 1, leader changed from <none> to e3d6e62aa5b546dbaeadd8ba934aa24c (127.0.98.1). New cstate: current_term: 1 leader_uuid: "e3d6e62aa5b546dbaeadd8ba934aa24c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20705812c28647ec9ef08248d5de0736" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42807 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "148385836b554faaba8ee69765e73730" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32885 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e3d6e62aa5b546dbaeadd8ba934aa24c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36587 } health_report { overall_health: HEALTHY } } }
I20260927 14:54:09.074478 15159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:54:09.070762 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59830 (local address 127.0.98.1:36587)
0927 14:54:09.071086 (+ 324us) server_negotiation.cc:206] Beginning negotiation
0927 14:54:09.071093 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 14:54:09.071104 (+ 11us) server_negotiation.cc:407] Connection header received
0927 14:54:09.071172 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:54:09.071174 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:54:09.071228 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:54:09.071357 (+ 129us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 14:54:09.071603 (+ 246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:54:09.072108 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:54:09.072947 (+ 839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:54:09.073230 (+ 283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:54:09.073338 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:54:09.073606 (+ 268us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 14:54:09.074042 (+ 436us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 14:54:09.074168 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0927 14:54:09.074237 (+ 69us) server_negotiation.cc:299] Negotiation successful
0927 14:54:09.074310 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":241,"thread_start_us":107,"threads_started":1}
I20260927 14:54:09.078351 15091 raft_consensus.cc:1261] T 44b03fc9151b4b4ab88f7d1c5d86d2a0 P 20705812c28647ec9ef08248d5de0736 [term 1 FOLLOWER]: Refusing update from remote peer e3d6e62aa5b546dbaeadd8ba934aa24c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 14:54:09.078353 14958 raft_consensus.cc:1261] T 44b03fc9151b4b4ab88f7d1c5d86d2a0 P 148385836b554faaba8ee69765e73730 [term 1 FOLLOWER]: Refusing update from remote peer e3d6e62aa5b546dbaeadd8ba934aa24c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 14:54:09.079133 14872 consensus_queue.cc:1035] T 44b03fc9151b4b4ab88f7d1c5d86d2a0 P e3d6e62aa5b546dbaeadd8ba934aa24c [LEADER]: Connected to new peer: Peer: permanent_uuid: "148385836b554faaba8ee69765e73730" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32885 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 14:54:09.079332 15157 consensus_queue.cc:1035] T 44b03fc9151b4b4ab88f7d1c5d86d2a0 P e3d6e62aa5b546dbaeadd8ba934aa24c [LEADER]: Connected to new peer: Peer: permanent_uuid: "20705812c28647ec9ef08248d5de0736" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42807 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 14:54:09.094467 14677 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:56868:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 14:54:09.094653 14677 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:56868:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 14:54:09.096917 14677 catalog_manager.cc:5782] T 00000000000000000000000000000000 P bbbbbbf5e7484aeea9277246d74d663b: Sending DeleteTablet for 3 replicas of tablet 44b03fc9151b4b4ab88f7d1c5d86d2a0
I20260927 14:54:09.097741 14799 tablet_service.cc:1514] Processing DeleteTablet for tablet 44b03fc9151b4b4ab88f7d1c5d86d2a0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 14:54:09 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:59816
I20260927 14:54:09.097779 14938 tablet_service.cc:1514] Processing DeleteTablet for tablet 44b03fc9151b4b4ab88f7d1c5d86d2a0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 14:54:09 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:38688
I20260927 14:54:09.097787 15071 tablet_service.cc:1514] Processing DeleteTablet for tablet 44b03fc9151b4b4ab88f7d1c5d86d2a0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 14:54:09 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:58552
I20260927 14:54:09.098124 15169 tablet_replica.cc:330] T 44b03fc9151b4b4ab88f7d1c5d86d2a0 P e3d6e62aa5b546dbaeadd8ba934aa24c: stopping tablet replica
I20260927 14:54:09.098228 15170 tablet_replica.cc:330] T 44b03fc9151b4b4ab88f7d1c5d86d2a0 P 148385836b554faaba8ee69765e73730: stopping tablet replica
I20260927 14:54:09.098294 15169 raft_consensus.cc:2229] T 44b03fc9151b4b4ab88f7d1c5d86d2a0 P e3d6e62aa5b546dbaeadd8ba934aa24c [term 1 LEADER]: Raft consensus shutting down.
I20260927 14:54:09.098511 15170 raft_consensus.cc:2229] T 44b03fc9151b4b4ab88f7d1c5d86d2a0 P 148385836b554faaba8ee69765e73730 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 14:54:09.098522 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6936m5/build/debug/bin/kudu with pid 14716
I20260927 14:54:09.098671 15169 raft_consensus.cc:2258] T 44b03fc9151b4b4ab88f7d1c5d86d2a0 P e3d6e62aa5b546dbaeadd8ba934aa24c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 14:54:09.100021 15170 raft_consensus.cc:2258] T 44b03fc9151b4b4ab88f7d1c5d86d2a0 P 148385836b554faaba8ee69765e73730 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 14:54:09.100152 15171 tablet_replica.cc:330] T 44b03fc9151b4b4ab88f7d1c5d86d2a0 P 20705812c28647ec9ef08248d5de0736: stopping tablet replica
I20260927 14:54:09.100474 15171 raft_consensus.cc:2229] T 44b03fc9151b4b4ab88f7d1c5d86d2a0 P 20705812c28647ec9ef08248d5de0736 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 14:54:09.100785 15171 raft_consensus.cc:2258] T 44b03fc9151b4b4ab88f7d1c5d86d2a0 P 20705812c28647ec9ef08248d5de0736 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 14:54:09.101614 15170 ts_tablet_manager.cc:1905] T 44b03fc9151b4b4ab88f7d1c5d86d2a0 P 148385836b554faaba8ee69765e73730: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 14:54:09.102263 15171 ts_tablet_manager.cc:1905] T 44b03fc9151b4b4ab88f7d1c5d86d2a0 P 20705812c28647ec9ef08248d5de0736: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 14:54:09.104344 15170 ts_tablet_manager.cc:1918] T 44b03fc9151b4b4ab88f7d1c5d86d2a0 P 148385836b554faaba8ee69765e73730: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 14:54:09.104458 15170 log.cc:1196] T 44b03fc9151b4b4ab88f7d1c5d86d2a0 P 148385836b554faaba8ee69765e73730: Deleting WAL directory at /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790520684602251-392-0/minicluster-data/ts-1/wal/wals/44b03fc9151b4b4ab88f7d1c5d86d2a0
I20260927 14:54:09.104790 15170 ts_tablet_manager.cc:1939] T 44b03fc9151b4b4ab88f7d1c5d86d2a0 P 148385836b554faaba8ee69765e73730: Deleting consensus metadata
W20260927 14:54:09.104784 14665 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:36587 (error 108)
I20260927 14:54:09.105158 15171 ts_tablet_manager.cc:1918] T 44b03fc9151b4b4ab88f7d1c5d86d2a0 P 20705812c28647ec9ef08248d5de0736: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 14:54:09.105486 15171 log.cc:1196] T 44b03fc9151b4b4ab88f7d1c5d86d2a0 P 20705812c28647ec9ef08248d5de0736: Deleting WAL directory at /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790520684602251-392-0/minicluster-data/ts-2/wal/wals/44b03fc9151b4b4ab88f7d1c5d86d2a0
I20260927 14:54:09.105777 15147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:54:09.105464 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:36587 (local address 127.0.0.1:59844)
0927 14:54:09.105592 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0927 14:54:09.105676 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:36587: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260927 14:54:09.105919 15171 ts_tablet_manager.cc:1939] T 44b03fc9151b4b4ab88f7d1c5d86d2a0 P 20705812c28647ec9ef08248d5de0736: Deleting consensus metadata
I20260927 14:54:09.105919 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6936m5/build/debug/bin/kudu with pid 14870
I20260927 14:54:09.106009 14662 catalog_manager.cc:4841] TS 148385836b554faaba8ee69765e73730 (127.0.98.2:32885): tablet 44b03fc9151b4b4ab88f7d1c5d86d2a0 (table test-table [id=2feaa6cc2b444560bcdbffd60d0956f6]) successfully deleted
I20260927 14:54:09.107061 14664 catalog_manager.cc:4841] TS 20705812c28647ec9ef08248d5de0736 (127.0.98.3:42807): tablet 44b03fc9151b4b4ab88f7d1c5d86d2a0 (table test-table [id=2feaa6cc2b444560bcdbffd60d0956f6]) successfully deleted
W20260927 14:54:09.107527 14665 catalog_manager.cc:4584] TS e3d6e62aa5b546dbaeadd8ba934aa24c (127.0.98.1:36587): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 44b03fc9151b4b4ab88f7d1c5d86d2a0: Network error: Client connection negotiation failed: client connection to 127.0.98.1:36587: connect: Connection refused (error 111)
I20260927 14:54:09.112339 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6936m5/build/debug/bin/kudu with pid 15007
I20260927 14:54:09.117403 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6936m5/build/debug/bin/kudu with pid 14645
I20260927 14:54:09.123808 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6936m5/build/debug/bin/kudu
/tmp/dist-test-task6936m5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790520684602251-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790520684602251-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790520684602251-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790520684602251-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40583
--webserver_interface=127.0.98.62
--webserver_port=42835
--keytab_file=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790520684602251-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45035
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:40583
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790520684602251-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790520684602251-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790520684602251-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 14:54:09.236248 15173 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 14:54:09.236506 15173 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 14:54:09.236546 15173 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 14:54:09.240491 15173 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 14:54:09.240548 15173 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 14:54:09.240567 15173 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 14:54:09.240584 15173 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 14:54:09.240599 15173 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 14:54:09.245842 15173 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45035
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790520684602251-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790520684602251-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:40583
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790520684602251-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40583
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790520684602251-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=42835
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15173
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790520684602251-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:15:17 UTC on ccycloud.quasar-ledxfe.root.comops.site
build id 4676134
I20260927 14:54:09.246742 15173 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 14:54:09.247642 15173 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 14:54:09.255913 15179 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 14:54:09.255913 15178 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 14:54:09.255927 15181 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 14:54:09.256151 15173 server_base.cc:1034] running on GCE node
I20260927 14:54:09.256868 15173 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 14:54:09.257936 15173 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 14:54:09.259104 15173 hybrid_clock.cc:648] HybridClock initialized: now 1790520849259093 us; error 41 us; skew 500 ppm
Sep 27 14:54:09 dist-test-slave-xk1g krb5kdc[14097](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790520849, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 14:54:09.261708 15173 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 14:54:09.262547 15173 webserver.cc:460] Webserver started at http://127.0.98.62:42835/ using document root <none> and password file <none>
I20260927 14:54:09.263041 15173 fs_manager.cc:359] Metadata directory not provided
I20260927 14:54:09.263108 15173 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 14:54:09.266729 15173 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260927 14:54:09.268806 15188 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 14:54:09.270125 15173 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 14:54:09.270257 15173 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790520684602251-392-0/minicluster-data/master-0/data,/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790520684602251-392-0/minicluster-data/master-0/wal
uuid: "bbbbbbf5e7484aeea9277246d74d663b"
format_stamp: "Formatted at 2026-09-27 14:54:07 on dist-test-slave-xk1g"
server_key: "9cfcd137779d48b5a67105aae8a2a984"
server_key_iv: "6a588433d7e816f740f0b7dd7afcfe18"
server_key_version: "encryptionkey@0"
I20260927 14:54:09.270777 15173 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790520684602251-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790520684602251-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790520684602251-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 14:54:09.288419 15173 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 14:54:09.291394 15173 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 14:54:09.291633 15173 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 14:54:09.298895 15173 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:40583
I20260927 14:54:09.299175 15240 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:40583 every 8 connection(s)
I20260927 14:54:09.300082 15173 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790520684602251-392-0/minicluster-data/master-0/data/info.pb
I20260927 14:54:09.304414 15241 sys_catalog.cc:263] Verifying existing consensus state
I20260927 14:54:09.305989 15241 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bbbbbbf5e7484aeea9277246d74d663b: Bootstrap starting.
I20260927 14:54:09.310145 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6936m5/build/debug/bin/kudu as pid 15173
I20260927 14:54:09.310540 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6936m5/build/debug/bin/kudu
/tmp/dist-test-task6936m5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790520684602251-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790520684602251-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790520684602251-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790520684602251-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:36587
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:40583
--webserver_port=35933
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790520684602251-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45035
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790520684602251-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790520684602251-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790520684602251-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 14:54:09.323117 15241 log.cc:824] T 00000000000000000000000000000000 P bbbbbbf5e7484aeea9277246d74d663b: Log is configured to *not* fsync() on all Append() calls
I20260927 14:54:09.334357 15241 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bbbbbbf5e7484aeea9277246d74d663b: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 14:54:09.334731 15241 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bbbbbbf5e7484aeea9277246d74d663b: Bootstrap complete.
I20260927 14:54:09.338722 15241 raft_consensus.cc:348] T 00000000000000000000000000000000 P bbbbbbf5e7484aeea9277246d74d663b [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bbbbbbf5e7484aeea9277246d74d663b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40583 } }
I20260927 14:54:09.339113 15241 raft_consensus.cc:729] T 00000000000000000000000000000000 P bbbbbbf5e7484aeea9277246d74d663b [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: bbbbbbf5e7484aeea9277246d74d663b, State: Initialized, Role: FOLLOWER
I20260927 14:54:09.339583 15241 consensus_queue.cc:260] T 00000000000000000000000000000000 P bbbbbbf5e7484aeea9277246d74d663b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bbbbbbf5e7484aeea9277246d74d663b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40583 } }
I20260927 14:54:09.339752 15241 raft_consensus.cc:388] T 00000000000000000000000000000000 P bbbbbbf5e7484aeea9277246d74d663b [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 14:54:09.339834 15241 raft_consensus.cc:482] T 00000000000000000000000000000000 P bbbbbbf5e7484aeea9277246d74d663b [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 14:54:09.339957 15241 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bbbbbbf5e7484aeea9277246d74d663b [term 2 FOLLOWER]: Advancing to term 3
I20260927 14:54:09.341835 15241 raft_consensus.cc:504] T 00000000000000000000000000000000 P bbbbbbf5e7484aeea9277246d74d663b [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bbbbbbf5e7484aeea9277246d74d663b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40583 } }
I20260927 14:54:09.342159 15241 leader_election.cc:302] T 00000000000000000000000000000000 P bbbbbbf5e7484aeea9277246d74d663b [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bbbbbbf5e7484aeea9277246d74d663b; no voters:
I20260927 14:54:09.342512 15241 leader_election.cc:288] T 00000000000000000000000000000000 P bbbbbbf5e7484aeea9277246d74d663b [CANDIDATE]: Term 3 election: Requested vote from peers
I20260927 14:54:09.342592 15245 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bbbbbbf5e7484aeea9277246d74d663b [term 3 FOLLOWER]: Leader election won for term 3
I20260927 14:54:09.342779 15245 raft_consensus.cc:686] T 00000000000000000000000000000000 P bbbbbbf5e7484aeea9277246d74d663b [term 3 LEADER]: Becoming Leader. State: Replica: bbbbbbf5e7484aeea9277246d74d663b, State: Running, Role: LEADER
I20260927 14:54:09.343060 15245 consensus_queue.cc:237] T 00000000000000000000000000000000 P bbbbbbf5e7484aeea9277246d74d663b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bbbbbbf5e7484aeea9277246d74d663b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40583 } }
I20260927 14:54:09.343694 15241 sys_catalog.cc:564] T 00000000000000000000000000000000 P bbbbbbf5e7484aeea9277246d74d663b [sys.catalog]: configured and running, proceeding with master startup.
I20260927 14:54:09.343968 15245 sys_catalog.cc:455] T 00000000000000000000000000000000 P bbbbbbf5e7484aeea9277246d74d663b [sys.catalog]: SysCatalogTable state changed. Reason: New leader bbbbbbf5e7484aeea9277246d74d663b. Latest consensus state: current_term: 3 leader_uuid: "bbbbbbf5e7484aeea9277246d74d663b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bbbbbbf5e7484aeea9277246d74d663b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40583 } } }
I20260927 14:54:09.344082 15245 sys_catalog.cc:458] T 00000000000000000000000000000000 P bbbbbbf5e7484aeea9277246d74d663b [sys.catalog]: This master's current role is: LEADER
I20260927 14:54:09.344436 15249 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 14:54:09.344599 15246 sys_catalog.cc:455] T 00000000000000000000000000000000 P bbbbbbf5e7484aeea9277246d74d663b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "bbbbbbf5e7484aeea9277246d74d663b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bbbbbbf5e7484aeea9277246d74d663b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40583 } } }
I20260927 14:54:09.344794 15246 sys_catalog.cc:458] T 00000000000000000000000000000000 P bbbbbbf5e7484aeea9277246d74d663b [sys.catalog]: This master's current role is: LEADER
I20260927 14:54:09.349496 15249 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 14:54:09.350140 15249 catalog_manager.cc:1259] Loaded cluster ID: 8c868f16380a4788ae11a60bca2797c4
I20260927 14:54:09.350212 15249 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260927 14:54:09.351213 15249 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f029ffc69ad google::LogMessage::Fail() at ??:0
@ 0x7f029ffc988a google::LogMessage::SendToLog() at ??:0
@ 0x7f029ffc64de google::LogMessage::Flush() at ??:0
@ 0x7f029ffc7a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f02aa905fdd _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_ at ??:0
@ 0x7f02aa906d2f kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f02aa903579 _ZZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvENKUlvE_clEv at ??:0
@ 0x7f02aa947640 _ZNSt17_Function_handlerIFvvEZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f02a22183d7 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f02a2218c83 _ZZN4kudu10ThreadPool12CreateThreadEvENKUlvE_clEv at ??:0
@ 0x7f02a221a3f2 _ZNSt17_Function_handlerIFvvEZN4kudu10ThreadPool12CreateThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f02a2209512 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f02a384d6db start_thread at ??:0
@ 0x7f029eac461f clone at ??:0
W20260927 14:54:09.443672 15243 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 14:54:09.443976 15243 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 14:54:09.444070 15243 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 14:54:09.448199 15243 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 14:54:09.448280 15243 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 14:54:09.448377 15243 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 14:54:09.453615 15243 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45035
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790520684602251-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790520684602251-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790520684602251-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:36587
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790520684602251-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=35933
--tserver_master_addrs=127.0.98.62:40583
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15243
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790520684602251-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:15:17 UTC on ccycloud.quasar-ledxfe.root.comops.site
build id 4676134
I20260927 14:54:09.454653 15243 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 14:54:09.455669 15243 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 14:54:09.465549 15283 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 14:54:09.465550 15282 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 14:54:09.465740 15243 server_base.cc:1034] running on GCE node
W20260927 14:54:09.465804 15285 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 14:54:09.466324 15243 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 14:54:09.467010 15243 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 14:54:09.468201 15243 hybrid_clock.cc:648] HybridClock initialized: now 1790520849468178 us; error 44 us; skew 500 ppm
Sep 27 14:54:09 dist-test-slave-xk1g krb5kdc[14097](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790520849, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 14:54:09.470873 15243 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 14:54:09.471792 15243 webserver.cc:460] Webserver started at http://127.0.98.1:35933/ using document root <none> and password file <none>
I20260927 14:54:09.472316 15243 fs_manager.cc:359] Metadata directory not provided
I20260927 14:54:09.472411 15243 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 14:54:09.476469 15243 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260927 14:54:09.478820 15292 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 14:54:09.480154 15243 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260927 14:54:09.480295 15243 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790520684602251-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790520684602251-392-0/minicluster-data/ts-0/wal
uuid: "e3d6e62aa5b546dbaeadd8ba934aa24c"
format_stamp: "Formatted at 2026-09-27 14:54:07 on dist-test-slave-xk1g"
server_key: "a793d8b41e7bfe4a80c054b66217298c"
server_key_iv: "b9974b52713fa99a8ccf2c7ea4c775e1"
server_key_version: "encryptionkey@0"
I20260927 14:54:09.480834 15243 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790520684602251-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790520684602251-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790520684602251-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 14:54:09.497653 15243 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 14:54:09.500994 15243 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 14:54:09.501230 15243 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 14:54:09.501828 15243 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 14:54:09.502918 15299 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260927 14:54:09.508450 15243 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260927 14:54:09.508524 15243 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.006s user 0.000s sys 0.000s
I20260927 14:54:09.508605 15243 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260927 14:54:09.510242 15243 ts_tablet_manager.cc:610] Registered 1 tablets
I20260927 14:54:09.510311 15243 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.001s sys 0.000s
I20260927 14:54:09.510589 15299 tablet_bootstrap.cc:492] T 44b03fc9151b4b4ab88f7d1c5d86d2a0 P e3d6e62aa5b546dbaeadd8ba934aa24c: Bootstrap starting.
I20260927 14:54:09.520808 15243 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:36587
I20260927 14:54:09.520879 15406 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:36587 every 8 connection(s)
I20260927 14:54:09.521858 15243 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790520684602251-392-0/minicluster-data/ts-0/data/info.pb
I20260927 14:54:09.527093 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6936m5/build/debug/bin/kudu as pid 15243
I20260927 14:54:09.527501 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6936m5/build/debug/bin/kudu
/tmp/dist-test-task6936m5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790520684602251-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790520684602251-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790520684602251-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790520684602251-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:32885
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:40583
--webserver_port=45923
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790520684602251-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45035
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790520684602251-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790520684602251-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790520684602251-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 14:54:09.532060 15299 log.cc:824] T 44b03fc9151b4b4ab88f7d1c5d86d2a0 P e3d6e62aa5b546dbaeadd8ba934aa24c: Log is configured to *not* fsync() on all Append() calls
I20260927 14:54:09.541041 15299 tablet_bootstrap.cc:492] T 44b03fc9151b4b4ab88f7d1c5d86d2a0 P e3d6e62aa5b546dbaeadd8ba934aa24c: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 14:54:09.541495 15299 tablet_bootstrap.cc:492] T 44b03fc9151b4b4ab88f7d1c5d86d2a0 P e3d6e62aa5b546dbaeadd8ba934aa24c: Bootstrap complete.
I20260927 14:54:09.542914 15299 ts_tablet_manager.cc:1397] T 44b03fc9151b4b4ab88f7d1c5d86d2a0 P e3d6e62aa5b546dbaeadd8ba934aa24c: Time spent bootstrapping tablet: real 0.033s user 0.026s sys 0.005s
I20260927 14:54:09.546137 15299 raft_consensus.cc:348] T 44b03fc9151b4b4ab88f7d1c5d86d2a0 P e3d6e62aa5b546dbaeadd8ba934aa24c [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20705812c28647ec9ef08248d5de0736" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42807 } } peers { permanent_uuid: "148385836b554faaba8ee69765e73730" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32885 } } peers { permanent_uuid: "e3d6e62aa5b546dbaeadd8ba934aa24c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36587 } }
I20260927 14:54:09.546685 15299 raft_consensus.cc:729] T 44b03fc9151b4b4ab88f7d1c5d86d2a0 P e3d6e62aa5b546dbaeadd8ba934aa24c [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: e3d6e62aa5b546dbaeadd8ba934aa24c, State: Initialized, Role: FOLLOWER
I20260927 14:54:09.547212 15299 consensus_queue.cc:260] T 44b03fc9151b4b4ab88f7d1c5d86d2a0 P e3d6e62aa5b546dbaeadd8ba934aa24c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20705812c28647ec9ef08248d5de0736" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42807 } } peers { permanent_uuid: "148385836b554faaba8ee69765e73730" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32885 } } peers { permanent_uuid: "e3d6e62aa5b546dbaeadd8ba934aa24c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36587 } }
I20260927 14:54:09.548249 15299 ts_tablet_manager.cc:1428] T 44b03fc9151b4b4ab88f7d1c5d86d2a0 P e3d6e62aa5b546dbaeadd8ba934aa24c: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
W20260927 14:54:09.582300 15409 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 14:54:09.523784 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40583 (local address 127.0.98.1:52429)
0927 14:54:09.524262 (+ 478us) negotiation.cc:107] Waiting for socket to connect
0927 14:54:09.524314 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0927 14:54:09.525158 (+ 844us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:54:09.581526 (+ 56368us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40583: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":280,"thread_start_us":146,"threads_started":1}
W20260927 14:54:09.582856 15407 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:40583 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:40583: Client connection negotiation failed: client connection to 127.0.98.62:40583: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260927 14:54:09.583585 15409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:54:09.583363 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40583 (local address 127.0.98.1:39649)
0927 14:54:09.583477 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0927 14:54:09.583534 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40583: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260927 14:54:09.584246 15409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:54:09.584049 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40583 (local address 127.0.98.1:45273)
0927 14:54:09.584147 (+ 98us) negotiation.cc:107] Waiting for socket to connect
0927 14:54:09.584196 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40583: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260927 14:54:09.584453 15407 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260927 14:54:09.642205 15410 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 14:54:09.642437 15410 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 14:54:09.642500 15410 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 14:54:09.646514 15410 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 14:54:09.646565 15410 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 14:54:09.646667 15410 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 14:54:09.651666 15410 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45035
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790520684602251-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790520684602251-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790520684602251-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:32885
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790520684602251-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=45923
--tserver_master_addrs=127.0.98.62:40583
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15410
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790520684602251-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:15:17 UTC on ccycloud.quasar-ledxfe.root.comops.site
build id 4676134
I20260927 14:54:09.652632 15410 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 14:54:09.653636 15410 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 14:54:09.662881 15418 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 14:54:09.662973 15410 server_base.cc:1034] running on GCE node
W20260927 14:54:09.662900 15420 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 14:54:09.662878 15417 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 14:54:09.663486 15410 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 14:54:09.664103 15410 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 14:54:09.665259 15410 hybrid_clock.cc:648] HybridClock initialized: now 1790520849665238 us; error 51 us; skew 500 ppm
Sep 27 14:54:09 dist-test-slave-xk1g krb5kdc[14097](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790520849, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 14:54:09.667922 15410 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 14:54:09.668823 15410 webserver.cc:460] Webserver started at http://127.0.98.2:45923/ using document root <none> and password file <none>
I20260927 14:54:09.669418 15410 fs_manager.cc:359] Metadata directory not provided
I20260927 14:54:09.669507 15410 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 14:54:09.673022 15410 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260927 14:54:09.674947 15427 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 14:54:09.676052 15410 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 14:54:09.676191 15410 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790520684602251-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790520684602251-392-0/minicluster-data/ts-1/wal
uuid: "148385836b554faaba8ee69765e73730"
format_stamp: "Formatted at 2026-09-27 14:54:07 on dist-test-slave-xk1g"
server_key: "bb1f0ed822f3151730bc37dd3e4db41f"
server_key_iv: "5b105ad590e22475019065d8d3694582"
server_key_version: "encryptionkey@0"
I20260927 14:54:09.676703 15410 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790520684602251-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790520684602251-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790520684602251-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 14:54:09.718813 15410 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 14:54:09.721915 15410 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 14:54:09.722124 15410 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 14:54:09.722779 15410 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 14:54:09.723589 15410 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 14:54:09.723635 15410 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 14:54:09.723711 15410 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 14:54:09.723752 15410 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 14:54:09.732776 15410 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:32885
I20260927 14:54:09.732800 15540 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:32885 every 8 connection(s)
I20260927 14:54:09.733812 15410 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790520684602251-392-0/minicluster-data/ts-1/data/info.pb
I20260927 14:54:09.736996 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6936m5/build/debug/bin/kudu as pid 15410
I20260927 14:54:09.737452 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6936m5/build/debug/bin/kudu
/tmp/dist-test-task6936m5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790520684602251-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790520684602251-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790520684602251-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790520684602251-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:42807
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:40583
--webserver_port=46651
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790520684602251-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45035
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790520684602251-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790520684602251-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790520684602251-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 14:54:09.737773 15543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:54:09.735804 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40583 (local address 127.0.98.2:41121)
0927 14:54:09.736657 (+ 853us) negotiation.cc:107] Waiting for socket to connect
0927 14:54:09.736804 (+ 147us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40583: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":664,"thread_start_us":519,"threads_started":1}
W20260927 14:54:09.738219 15541 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:40583 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:40583: Client connection negotiation failed: client connection to 127.0.98.62:40583: connect: Connection refused (error 111)
I20260927 14:54:09.738840 15543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:54:09.738559 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40583 (local address 127.0.98.2:46035)
0927 14:54:09.738696 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0927 14:54:09.738759 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40583: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260927 14:54:09.739532 15543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:54:09.739321 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40583 (local address 127.0.98.2:44419)
0927 14:54:09.739419 (+ 98us) negotiation.cc:107] Waiting for socket to connect
0927 14:54:09.739470 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40583: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260927 14:54:09.739760 15541 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260927 14:54:09.847325 15544 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 14:54:09.847574 15544 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 14:54:09.847618 15544 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 14:54:09.851632 15544 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 14:54:09.851696 15544 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 14:54:09.851814 15544 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 14:54:09.856979 15544 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45035
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790520684602251-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790520684602251-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790520684602251-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:42807
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790520684602251-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=46651
--tserver_master_addrs=127.0.98.62:40583
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15544
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790520684602251-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:15:17 UTC on ccycloud.quasar-ledxfe.root.comops.site
build id 4676134
I20260927 14:54:09.858057 15544 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 14:54:09.859076 15544 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 14:54:09.868584 15549 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 14:54:09.868602 15550 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 14:54:09.868662 15544 server_base.cc:1034] running on GCE node
W20260927 14:54:09.868584 15552 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 14:54:09.869290 15544 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 14:54:09.869902 15544 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 14:54:09.871155 15544 hybrid_clock.cc:648] HybridClock initialized: now 1790520849871123 us; error 44 us; skew 500 ppm
Sep 27 14:54:09 dist-test-slave-xk1g krb5kdc[14097](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790520849, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 14:54:09.873857 15544 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 14:54:09.874656 15544 webserver.cc:460] Webserver started at http://127.0.98.3:46651/ using document root <none> and password file <none>
I20260927 14:54:09.875162 15544 fs_manager.cc:359] Metadata directory not provided
I20260927 14:54:09.875223 15544 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 14:54:09.878733 15544 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260927 14:54:09.880681 15559 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 14:54:09.881803 15544 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 14:54:09.881887 15544 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790520684602251-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790520684602251-392-0/minicluster-data/ts-2/wal
uuid: "20705812c28647ec9ef08248d5de0736"
format_stamp: "Formatted at 2026-09-27 14:54:07 on dist-test-slave-xk1g"
server_key: "b8a0672c73a73a48a6e0652b1e6efab0"
server_key_iv: "b8386891c67ea1e1632665d2d00fb3dd"
server_key_version: "encryptionkey@0"
I20260927 14:54:09.882367 15544 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790520684602251-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790520684602251-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790520684602251-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 14:54:09.900698 15544 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 14:54:09.903537 15544 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 14:54:09.903743 15544 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 14:54:09.904289 15544 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 14:54:09.905019 15544 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 14:54:09.905128 15544 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 14:54:09.905192 15544 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 14:54:09.905210 15544 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 14:54:09.913513 15544 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:42807
I20260927 14:54:09.913537 15672 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:42807 every 8 connection(s)
I20260927 14:54:09.914556 15544 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790520684602251-392-0/minicluster-data/ts-2/data/info.pb
I20260927 14:54:09.918210 15675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:54:09.916709 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40583 (local address 127.0.98.3:56837)
0927 14:54:09.917426 (+ 717us) negotiation.cc:107] Waiting for socket to connect
0927 14:54:09.917529 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40583: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":533,"thread_start_us":109,"threads_started":1}
W20260927 14:54:09.918670 15673 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:40583 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:40583: Client connection negotiation failed: client connection to 127.0.98.62:40583: connect: Connection refused (error 111)
I20260927 14:54:09.919340 15675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:54:09.919026 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40583 (local address 127.0.98.3:37669)
0927 14:54:09.919238 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0927 14:54:09.919287 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40583: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":128}
I20260927 14:54:09.920020 15675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:54:09.919786 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40583 (local address 127.0.98.3:46179)
0927 14:54:09.919887 (+ 101us) negotiation.cc:107] Waiting for socket to connect
0927 14:54:09.919939 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40583: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260927 14:54:09.920202 15673 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260927 14:54:09.924160 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6936m5/build/debug/bin/kudu as pid 15544
I20260927 14:54:10.585508 15677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:54:10.585103 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40583 (local address 127.0.98.1:47763)
0927 14:54:10.585362 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0927 14:54:10.585412 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40583: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":78,"threads_started":1}
I20260927 14:54:10.740741 15678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:54:10.740378 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40583 (local address 127.0.98.2:41885)
0927 14:54:10.740594 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0927 14:54:10.740644 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40583: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":153,"thread_start_us":69,"threads_started":1}
I20260927 14:54:10.921197 15679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:54:10.920689 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40583 (local address 127.0.98.3:49085)
0927 14:54:10.920984 (+ 295us) negotiation.cc:107] Waiting for socket to connect
0927 14:54:10.921037 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40583: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":228,"thread_start_us":64,"threads_started":1}
I20260927 14:54:10.980342 15680 raft_consensus.cc:482] T 44b03fc9151b4b4ab88f7d1c5d86d2a0 P e3d6e62aa5b546dbaeadd8ba934aa24c [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 14:54:10.980547 15680 raft_consensus.cc:504] T 44b03fc9151b4b4ab88f7d1c5d86d2a0 P e3d6e62aa5b546dbaeadd8ba934aa24c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20705812c28647ec9ef08248d5de0736" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42807 } } peers { permanent_uuid: "148385836b554faaba8ee69765e73730" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32885 } } peers { permanent_uuid: "e3d6e62aa5b546dbaeadd8ba934aa24c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36587 } }
I20260927 14:54:10.981671 15680 leader_election.cc:288] T 44b03fc9151b4b4ab88f7d1c5d86d2a0 P e3d6e62aa5b546dbaeadd8ba934aa24c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 20705812c28647ec9ef08248d5de0736 (127.0.98.3:42807), 148385836b554faaba8ee69765e73730 (127.0.98.2:32885)
Sep 27 14:54:10 dist-test-slave-xk1g krb5kdc[14097](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790520849, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.3@KRBTEST.COM
Sep 27 14:54:10 dist-test-slave-xk1g krb5kdc[14097](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790520849, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.2@KRBTEST.COM
I20260927 14:54:10.999157 15682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:54:10.982091 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:39903 (local address 127.0.98.2:32885)
0927 14:54:10.982489 (+ 398us) server_negotiation.cc:206] Beginning negotiation
0927 14:54:10.982514 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0927 14:54:10.982559 (+ 45us) server_negotiation.cc:407] Connection header received
0927 14:54:10.983460 (+ 901us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:54:10.983468 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:54:10.983947 (+ 479us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:54:10.984295 (+ 348us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:54:10.986124 (+ 1829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:54:10.987725 (+ 1601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:54:10.989725 (+ 2000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:54:10.990462 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:54:10.990637 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:54:10.994263 (+ 3626us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:54:10.994269 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:54:10.994281 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 14:54:10.994312 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:54:10.996361 (+ 2049us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:54:10.996923 (+ 562us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:54:10.996930 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:54:10.996936 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:54:10.997110 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:54:10.997558 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:54:10.997561 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:54:10.997563 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:54:10.997998 (+ 435us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:54:10.998148 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0927 14:54:10.998705 (+ 557us) server_negotiation.cc:299] Negotiation successful
0927 14:54:10.998958 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":283,"thread_start_us":69,"threads_started":1}
I20260927 14:54:10.999167 15677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:54:10.981942 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:32885 (local address 127.0.98.1:39903)
0927 14:54:10.982174 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0927 14:54:10.982191 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 14:54:10.982307 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:54:10.984409 (+ 2102us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:54:10.984416 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:54:10.985076 (+ 660us) client_negotiation.cc:189] Negotiated authn=SASL
0927 14:54:10.985926 (+ 850us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:54:10.985949 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:54:10.987900 (+ 1951us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:54:10.987902 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:54:10.989591 (+ 1689us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:54:10.989603 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:54:10.990600 (+ 997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:54:10.990604 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:54:10.990892 (+ 288us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:54:10.991712 (+ 820us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 14:54:10.991744 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0927 14:54:10.994055 (+ 2311us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 14:54:10.996488 (+ 2433us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 14:54:10.996492 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 14:54:10.996494 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 14:54:10.996792 (+ 298us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 14:54:10.997300 (+ 508us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 14:54:10.997303 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 14:54:10.997305 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 14:54:10.997442 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 14:54:10.998120 (+ 678us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 14:54:10.998127 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 14:54:10.998373 (+ 246us) client_negotiation.cc:769] Sending connection context
0927 14:54:10.998564 (+ 191us) client_negotiation.cc:240] Negotiation successful
0927 14:54:10.998940 (+ 376us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":133,"mutex_wait_us":385,"spinlock_wait_cycles":45952}
I20260927 14:54:10.999226 15681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:54:10.981942 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:42807 (local address 127.0.98.1:47691)
0927 14:54:10.982266 (+ 324us) negotiation.cc:107] Waiting for socket to connect
0927 14:54:10.982278 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 14:54:10.982352 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:54:10.984534 (+ 2182us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:54:10.984537 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:54:10.985076 (+ 539us) client_negotiation.cc:189] Negotiated authn=SASL
0927 14:54:10.985926 (+ 850us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:54:10.985964 (+ 38us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:54:10.987897 (+ 1933us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:54:10.987901 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:54:10.989591 (+ 1690us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:54:10.989603 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:54:10.990600 (+ 997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:54:10.990604 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:54:10.990891 (+ 287us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:54:10.991712 (+ 821us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 14:54:10.991740 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0927 14:54:10.993841 (+ 2101us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 14:54:10.996407 (+ 2566us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 14:54:10.996414 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 14:54:10.996430 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0927 14:54:10.996798 (+ 368us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 14:54:10.997270 (+ 472us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 14:54:10.997273 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 14:54:10.997275 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 14:54:10.997452 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 14:54:10.998120 (+ 668us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 14:54:10.998127 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 14:54:10.998373 (+ 246us) client_negotiation.cc:769] Sending connection context
0927 14:54:10.998563 (+ 190us) client_negotiation.cc:240] Negotiation successful
0927 14:54:10.999026 (+ 463us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":262,"mutex_wait_us":585,"spinlock_wait_cycles":44928,"thread_start_us":63,"threads_started":1}
I20260927 14:54:10.999184 15683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:54:10.982091 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:47691 (local address 127.0.98.3:42807)
0927 14:54:10.982489 (+ 398us) server_negotiation.cc:206] Beginning negotiation
0927 14:54:10.982514 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0927 14:54:10.982559 (+ 45us) server_negotiation.cc:407] Connection header received
0927 14:54:10.983460 (+ 901us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:54:10.983468 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:54:10.983947 (+ 479us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:54:10.984295 (+ 348us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:54:10.986124 (+ 1829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:54:10.987723 (+ 1599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:54:10.989722 (+ 1999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:54:10.990462 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:54:10.990637 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:54:10.994112 (+ 3475us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:54:10.994117 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:54:10.994131 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 14:54:10.994164 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:54:10.996265 (+ 2101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:54:10.996934 (+ 669us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:54:10.996939 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:54:10.996944 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:54:10.997113 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:54:10.997560 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:54:10.997563 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:54:10.997565 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:54:10.997998 (+ 433us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:54:10.998148 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0927 14:54:10.998690 (+ 542us) server_negotiation.cc:299] Negotiation successful
0927 14:54:10.998981 (+ 291us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":286,"thread_start_us":118,"threads_started":1}
W20260927 14:54:11.001750 15295 leader_election.cc:341] T 44b03fc9151b4b4ab88f7d1c5d86d2a0 P e3d6e62aa5b546dbaeadd8ba934aa24c [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 20705812c28647ec9ef08248d5de0736 (127.0.98.3:42807): Not found: Tablet not found: 44b03fc9151b4b4ab88f7d1c5d86d2a0
W20260927 14:54:11.001927 15293 leader_election.cc:341] T 44b03fc9151b4b4ab88f7d1c5d86d2a0 P e3d6e62aa5b546dbaeadd8ba934aa24c [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 148385836b554faaba8ee69765e73730 (127.0.98.2:32885): Not found: Tablet not found: 44b03fc9151b4b4ab88f7d1c5d86d2a0
I20260927 14:54:11.002007 15293 leader_election.cc:302] T 44b03fc9151b4b4ab88f7d1c5d86d2a0 P e3d6e62aa5b546dbaeadd8ba934aa24c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e3d6e62aa5b546dbaeadd8ba934aa24c; no voters: 148385836b554faaba8ee69765e73730, 20705812c28647ec9ef08248d5de0736
I20260927 14:54:11.002281 15680 raft_consensus.cc:2726] T 44b03fc9151b4b4ab88f7d1c5d86d2a0 P e3d6e62aa5b546dbaeadd8ba934aa24c [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 14:54:11.586616 15684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:54:11.586170 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40583 (local address 127.0.98.1:59473)
0927 14:54:11.586425 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0927 14:54:11.586521 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40583: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":85,"threads_started":1}
I20260927 14:54:11.741851 15685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:54:11.741476 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40583 (local address 127.0.98.2:33039)
0927 14:54:11.741700 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0927 14:54:11.741753 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40583: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":67,"threads_started":1}
I20260927 14:54:11.922181 15686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:54:11.921814 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40583 (local address 127.0.98.3:45495)
0927 14:54:11.922053 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0927 14:54:11.922102 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40583: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":79,"threads_started":1}
I20260927 14:54:12.587774 15687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:54:12.587359 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40583 (local address 127.0.98.1:48161)
0927 14:54:12.587619 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0927 14:54:12.587679 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40583: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":74,"threads_started":1}
I20260927 14:54:12.655486 15688 raft_consensus.cc:482] T 44b03fc9151b4b4ab88f7d1c5d86d2a0 P e3d6e62aa5b546dbaeadd8ba934aa24c [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 14:54:12.655640 15688 raft_consensus.cc:504] T 44b03fc9151b4b4ab88f7d1c5d86d2a0 P e3d6e62aa5b546dbaeadd8ba934aa24c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20705812c28647ec9ef08248d5de0736" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42807 } } peers { permanent_uuid: "148385836b554faaba8ee69765e73730" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32885 } } peers { permanent_uuid: "e3d6e62aa5b546dbaeadd8ba934aa24c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36587 } }
I20260927 14:54:12.656164 15688 leader_election.cc:288] T 44b03fc9151b4b4ab88f7d1c5d86d2a0 P e3d6e62aa5b546dbaeadd8ba934aa24c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 20705812c28647ec9ef08248d5de0736 (127.0.98.3:42807), 148385836b554faaba8ee69765e73730 (127.0.98.2:32885)
W20260927 14:54:12.656893 15293 leader_election.cc:341] T 44b03fc9151b4b4ab88f7d1c5d86d2a0 P e3d6e62aa5b546dbaeadd8ba934aa24c [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 148385836b554faaba8ee69765e73730 (127.0.98.2:32885): Not found: Tablet not found: 44b03fc9151b4b4ab88f7d1c5d86d2a0
W20260927 14:54:12.656981 15295 leader_election.cc:341] T 44b03fc9151b4b4ab88f7d1c5d86d2a0 P e3d6e62aa5b546dbaeadd8ba934aa24c [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 20705812c28647ec9ef08248d5de0736 (127.0.98.3:42807): Not found: Tablet not found: 44b03fc9151b4b4ab88f7d1c5d86d2a0
I20260927 14:54:12.657122 15295 leader_election.cc:302] T 44b03fc9151b4b4ab88f7d1c5d86d2a0 P e3d6e62aa5b546dbaeadd8ba934aa24c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e3d6e62aa5b546dbaeadd8ba934aa24c; no voters: 148385836b554faaba8ee69765e73730, 20705812c28647ec9ef08248d5de0736
I20260927 14:54:12.657274 15688 raft_consensus.cc:2726] T 44b03fc9151b4b4ab88f7d1c5d86d2a0 P e3d6e62aa5b546dbaeadd8ba934aa24c [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 14:54:12.742895 15689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:54:12.742537 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40583 (local address 127.0.98.2:43067)
0927 14:54:12.742758 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0927 14:54:12.742808 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40583: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":69,"threads_started":1}
I20260927 14:54:12.923336 15690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:54:12.922918 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40583 (local address 127.0.98.3:34031)
0927 14:54:12.923176 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0927 14:54:12.923239 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40583: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":104,"threads_started":1}
I20260927 14:54:13.588878 15691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:54:13.588462 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40583 (local address 127.0.98.1:41101)
0927 14:54:13.588721 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0927 14:54:13.588780 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40583: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":69,"threads_started":1}
I20260927 14:54:13.744042 15692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:54:13.743626 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40583 (local address 127.0.98.2:43161)
0927 14:54:13.743894 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0927 14:54:13.743946 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40583: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":202,"thread_start_us":84,"threads_started":1}
I20260927 14:54:13.924386 15693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:54:13.923968 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40583 (local address 127.0.98.3:46247)
0927 14:54:13.924240 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0927 14:54:13.924295 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40583: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":83,"threads_started":1}
I20260927 14:54:14.590217 15694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:54:14.589782 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40583 (local address 127.0.98.1:57011)
0927 14:54:14.590068 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0927 14:54:14.590124 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40583: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":215,"thread_start_us":78,"threads_started":1}
I20260927 14:54:14.745246 15695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:54:14.744775 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40583 (local address 127.0.98.2:33553)
0927 14:54:14.745040 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0927 14:54:14.745154 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40583: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":76,"threads_started":1}
I20260927 14:54:14.925583 15696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:54:14.925214 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40583 (local address 127.0.98.3:39277)
0927 14:54:14.925447 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0927 14:54:14.925497 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40583: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":168,"thread_start_us":82,"threads_started":1}
I20260927 14:54:14.943708 15697 raft_consensus.cc:482] T 44b03fc9151b4b4ab88f7d1c5d86d2a0 P e3d6e62aa5b546dbaeadd8ba934aa24c [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 14:54:14.943838 15697 raft_consensus.cc:504] T 44b03fc9151b4b4ab88f7d1c5d86d2a0 P e3d6e62aa5b546dbaeadd8ba934aa24c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20705812c28647ec9ef08248d5de0736" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42807 } } peers { permanent_uuid: "148385836b554faaba8ee69765e73730" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32885 } } peers { permanent_uuid: "e3d6e62aa5b546dbaeadd8ba934aa24c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36587 } }
I20260927 14:54:14.944200 15697 leader_election.cc:288] T 44b03fc9151b4b4ab88f7d1c5d86d2a0 P e3d6e62aa5b546dbaeadd8ba934aa24c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 20705812c28647ec9ef08248d5de0736 (127.0.98.3:42807), 148385836b554faaba8ee69765e73730 (127.0.98.2:32885)
W20260927 14:54:14.944953 15295 leader_election.cc:341] T 44b03fc9151b4b4ab88f7d1c5d86d2a0 P e3d6e62aa5b546dbaeadd8ba934aa24c [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 20705812c28647ec9ef08248d5de0736 (127.0.98.3:42807): Not found: Tablet not found: 44b03fc9151b4b4ab88f7d1c5d86d2a0
W20260927 14:54:14.945040 15293 leader_election.cc:341] T 44b03fc9151b4b4ab88f7d1c5d86d2a0 P e3d6e62aa5b546dbaeadd8ba934aa24c [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 148385836b554faaba8ee69765e73730 (127.0.98.2:32885): Not found: Tablet not found: 44b03fc9151b4b4ab88f7d1c5d86d2a0
I20260927 14:54:14.945178 15293 leader_election.cc:302] T 44b03fc9151b4b4ab88f7d1c5d86d2a0 P e3d6e62aa5b546dbaeadd8ba934aa24c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e3d6e62aa5b546dbaeadd8ba934aa24c; no voters: 148385836b554faaba8ee69765e73730, 20705812c28647ec9ef08248d5de0736
I20260927 14:54:14.945363 15697 raft_consensus.cc:2726] T 44b03fc9151b4b4ab88f7d1c5d86d2a0 P e3d6e62aa5b546dbaeadd8ba934aa24c [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 14:54:15.591620 15698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:54:15.591110 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40583 (local address 127.0.98.1:34023)
0927 14:54:15.591409 (+ 299us) negotiation.cc:107] Waiting for socket to connect
0927 14:54:15.591482 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40583: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":70,"threads_started":1}
I20260927 14:54:15.746338 15699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:54:15.745949 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40583 (local address 127.0.98.2:33355)
0927 14:54:15.746197 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0927 14:54:15.746251 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40583: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":89,"threads_started":1}
I20260927 14:54:15.926829 15700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:54:15.926414 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40583 (local address 127.0.98.3:60521)
0927 14:54:15.926686 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0927 14:54:15.926744 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40583: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":199,"thread_start_us":73,"threads_started":1}
I20260927 14:54:16.592803 15701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:54:16.592380 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40583 (local address 127.0.98.1:33355)
0927 14:54:16.592643 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0927 14:54:16.592700 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40583: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":75,"threads_started":1}
I20260927 14:54:16.674520 15702 raft_consensus.cc:482] T 44b03fc9151b4b4ab88f7d1c5d86d2a0 P e3d6e62aa5b546dbaeadd8ba934aa24c [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 14:54:16.674649 15702 raft_consensus.cc:504] T 44b03fc9151b4b4ab88f7d1c5d86d2a0 P e3d6e62aa5b546dbaeadd8ba934aa24c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20705812c28647ec9ef08248d5de0736" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42807 } } peers { permanent_uuid: "148385836b554faaba8ee69765e73730" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32885 } } peers { permanent_uuid: "e3d6e62aa5b546dbaeadd8ba934aa24c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36587 } }
I20260927 14:54:16.675072 15702 leader_election.cc:288] T 44b03fc9151b4b4ab88f7d1c5d86d2a0 P e3d6e62aa5b546dbaeadd8ba934aa24c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 20705812c28647ec9ef08248d5de0736 (127.0.98.3:42807), 148385836b554faaba8ee69765e73730 (127.0.98.2:32885)
W20260927 14:54:16.675823 15295 leader_election.cc:341] T 44b03fc9151b4b4ab88f7d1c5d86d2a0 P e3d6e62aa5b546dbaeadd8ba934aa24c [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 20705812c28647ec9ef08248d5de0736 (127.0.98.3:42807): Not found: Tablet not found: 44b03fc9151b4b4ab88f7d1c5d86d2a0
W20260927 14:54:16.675910 15293 leader_election.cc:341] T 44b03fc9151b4b4ab88f7d1c5d86d2a0 P e3d6e62aa5b546dbaeadd8ba934aa24c [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 148385836b554faaba8ee69765e73730 (127.0.98.2:32885): Not found: Tablet not found: 44b03fc9151b4b4ab88f7d1c5d86d2a0
I20260927 14:54:16.676019 15293 leader_election.cc:302] T 44b03fc9151b4b4ab88f7d1c5d86d2a0 P e3d6e62aa5b546dbaeadd8ba934aa24c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e3d6e62aa5b546dbaeadd8ba934aa24c; no voters: 148385836b554faaba8ee69765e73730, 20705812c28647ec9ef08248d5de0736
I20260927 14:54:16.676172 15702 raft_consensus.cc:2726] T 44b03fc9151b4b4ab88f7d1c5d86d2a0 P e3d6e62aa5b546dbaeadd8ba934aa24c [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 14:54:16.747496 15703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:54:16.747028 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40583 (local address 127.0.98.2:42995)
0927 14:54:16.747353 (+ 325us) negotiation.cc:107] Waiting for socket to connect
0927 14:54:16.747407 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40583: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":254,"thread_start_us":85,"threads_started":1}
I20260927 14:54:16.927949 15704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:54:16.927525 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40583 (local address 127.0.98.3:37335)
0927 14:54:16.927812 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0927 14:54:16.927862 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40583: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":224,"thread_start_us":78,"threads_started":1}
I20260927 14:54:17.593919 15705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:54:17.593516 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40583 (local address 127.0.98.1:38863)
0927 14:54:17.593771 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0927 14:54:17.593827 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40583: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":82,"threads_started":1}
I20260927 14:54:17.748586 15706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:54:17.748209 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40583 (local address 127.0.98.2:47115)
0927 14:54:17.748447 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0927 14:54:17.748498 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40583: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":172,"thread_start_us":84,"threads_started":1}
I20260927 14:54:17.929194 15707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:54:17.928738 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40583 (local address 127.0.98.3:45271)
0927 14:54:17.928984 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0927 14:54:17.929039 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40583: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":71,"threads_started":1}
I20260927 14:54:18.595095 15708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:54:18.594680 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40583 (local address 127.0.98.1:46363)
0927 14:54:18.594949 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0927 14:54:18.595002 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40583: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":102,"threads_started":1}
I20260927 14:54:18.749660 15709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:54:18.749301 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40583 (local address 127.0.98.2:46039)
0927 14:54:18.749522 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0927 14:54:18.749574 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40583: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":154,"thread_start_us":68,"threads_started":1}
I20260927 14:54:18.930369 15710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:54:18.929939 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40583 (local address 127.0.98.3:52181)
0927 14:54:18.930220 (+ 281us) negotiation.cc:107] Waiting for socket to connect
0927 14:54:18.930279 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40583: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":217,"thread_start_us":120,"threads_started":1}
I20260927 14:54:19.596451 15711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:54:19.595989 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40583 (local address 127.0.98.1:41367)
0927 14:54:19.596291 (+ 302us) negotiation.cc:107] Waiting for socket to connect
0927 14:54:19.596348 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40583: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":226,"thread_start_us":110,"threads_started":1}
I20260927 14:54:19.750758 15712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:54:19.750346 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40583 (local address 127.0.98.2:35481)
0927 14:54:19.750604 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0927 14:54:19.750670 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40583: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":162,"thread_start_us":66,"threads_started":1}
I20260927 14:54:19.931522 15713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:54:19.931059 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40583 (local address 127.0.98.3:54007)
0927 14:54:19.931379 (+ 320us) negotiation.cc:107] Waiting for socket to connect
0927 14:54:19.931434 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40583: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":250,"thread_start_us":69,"threads_started":1}
I20260927 14:54:20.597829 15714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:54:20.597388 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40583 (local address 127.0.98.1:53287)
0927 14:54:20.597674 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0927 14:54:20.597735 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40583: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":207,"thread_start_us":84,"threads_started":1}
I20260927 14:54:20.751977 15715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:54:20.751538 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40583 (local address 127.0.98.2:55277)
0927 14:54:20.751825 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0927 14:54:20.751886 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40583: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":213,"thread_start_us":92,"threads_started":1}
I20260927 14:54:20.932750 15716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:54:20.932278 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40583 (local address 127.0.98.3:40135)
0927 14:54:20.932548 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0927 14:54:20.932605 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40583: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":88,"threads_started":1}
I20260927 14:54:21.599211 15717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:54:21.598743 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40583 (local address 127.0.98.1:37281)
0927 14:54:21.599021 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0927 14:54:21.599086 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40583: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":180,"thread_start_us":94,"threads_started":1}
I20260927 14:54:21.753187 15718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:54:21.752662 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40583 (local address 127.0.98.2:54671)
0927 14:54:21.752946 (+ 284us) negotiation.cc:107] Waiting for socket to connect
0927 14:54:21.753006 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40583: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":214,"thread_start_us":88,"threads_started":1}
I20260927 14:54:21.933920 15719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:54:21.933514 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40583 (local address 127.0.98.3:48319)
0927 14:54:21.933766 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0927 14:54:21.933826 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40583: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":175,"thread_start_us":70,"threads_started":1}
I20260927 14:54:22.600505 15720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:54:22.600032 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40583 (local address 127.0.98.1:60557)
0927 14:54:22.600300 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0927 14:54:22.600385 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40583: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":85,"threads_started":1}
I20260927 14:54:22.754311 15721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:54:22.753930 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40583 (local address 127.0.98.2:41881)
0927 14:54:22.754161 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0927 14:54:22.754219 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40583: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":168,"thread_start_us":74,"threads_started":1}
I20260927 14:54:22.935107 15722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:54:22.934660 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40583 (local address 127.0.98.3:41005)
0927 14:54:22.934936 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0927 14:54:22.935011 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40583: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":71,"threads_started":1}
I20260927 14:54:23.601761 15723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:54:23.601340 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40583 (local address 127.0.98.1:45575)
0927 14:54:23.601618 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0927 14:54:23.601670 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40583: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":204,"thread_start_us":72,"threads_started":1}
I20260927 14:54:23.755575 15724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:54:23.755092 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40583 (local address 127.0.98.2:40213)
0927 14:54:23.755372 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0927 14:54:23.755429 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40583: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":201,"thread_start_us":80,"threads_started":1}
I20260927 14:54:23.936401 15725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:54:23.935881 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40583 (local address 127.0.98.3:38733)
0927 14:54:23.936169 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0927 14:54:23.936259 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40583: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":91,"threads_started":1}
I20260927 14:54:23.936444 15726 raft_consensus.cc:482] T 44b03fc9151b4b4ab88f7d1c5d86d2a0 P e3d6e62aa5b546dbaeadd8ba934aa24c [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 14:54:23.936656 15726 raft_consensus.cc:504] T 44b03fc9151b4b4ab88f7d1c5d86d2a0 P e3d6e62aa5b546dbaeadd8ba934aa24c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20705812c28647ec9ef08248d5de0736" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42807 } } peers { permanent_uuid: "148385836b554faaba8ee69765e73730" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32885 } } peers { permanent_uuid: "e3d6e62aa5b546dbaeadd8ba934aa24c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36587 } }
I20260927 14:54:23.937119 15726 leader_election.cc:288] T 44b03fc9151b4b4ab88f7d1c5d86d2a0 P e3d6e62aa5b546dbaeadd8ba934aa24c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 20705812c28647ec9ef08248d5de0736 (127.0.98.3:42807), 148385836b554faaba8ee69765e73730 (127.0.98.2:32885)
W20260927 14:54:23.937840 15295 leader_election.cc:341] T 44b03fc9151b4b4ab88f7d1c5d86d2a0 P e3d6e62aa5b546dbaeadd8ba934aa24c [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 20705812c28647ec9ef08248d5de0736 (127.0.98.3:42807): Not found: Tablet not found: 44b03fc9151b4b4ab88f7d1c5d86d2a0
W20260927 14:54:23.937929 15293 leader_election.cc:341] T 44b03fc9151b4b4ab88f7d1c5d86d2a0 P e3d6e62aa5b546dbaeadd8ba934aa24c [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 148385836b554faaba8ee69765e73730 (127.0.98.2:32885): Not found: Tablet not found: 44b03fc9151b4b4ab88f7d1c5d86d2a0
I20260927 14:54:23.937988 15293 leader_election.cc:302] T 44b03fc9151b4b4ab88f7d1c5d86d2a0 P e3d6e62aa5b546dbaeadd8ba934aa24c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e3d6e62aa5b546dbaeadd8ba934aa24c; no voters: 148385836b554faaba8ee69765e73730, 20705812c28647ec9ef08248d5de0736
I20260927 14:54:23.938131 15726 raft_consensus.cc:2726] T 44b03fc9151b4b4ab88f7d1c5d86d2a0 P e3d6e62aa5b546dbaeadd8ba934aa24c [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 14:54:24.603070 15727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:54:24.602669 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40583 (local address 127.0.98.1:56847)
0927 14:54:24.602924 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0927 14:54:24.602978 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40583: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":83,"threads_started":1}
I20260927 14:54:24.756911 15728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:54:24.756441 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40583 (local address 127.0.98.2:54947)
0927 14:54:24.756758 (+ 317us) negotiation.cc:107] Waiting for socket to connect
0927 14:54:24.756817 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40583: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":234,"thread_start_us":87,"threads_started":1}
I20260927 14:54:24.932752 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6936m5/build/debug/bin/kudu with pid 15243
I20260927 14:54:24.937861 15729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:54:24.937459 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40583 (local address 127.0.98.3:53891)
0927 14:54:24.937700 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0927 14:54:24.937767 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40583: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":163,"thread_start_us":90,"threads_started":1}
I20260927 14:54:24.938602 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6936m5/build/debug/bin/kudu with pid 15410
I20260927 14:54:24.943928 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6936m5/build/debug/bin/kudu with pid 15544
2026-09-27T14:54:24Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (19231 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790520684602251-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 14:54:25 dist-test-slave-xk1g krb5kdc[15732](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 14:54:25 dist-test-slave-xk1g krb5kdc[15732](info): set up 2 sockets
Sep 27 14:54:25 dist-test-slave-xk1g krb5kdc[15732](info): commencing operation
krb5kdc: starting...
W20260927 14:54:25.989375 392 mini_kdc.cc:121] Time spent starting KDC: real 1.024s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790520684602251-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790520684602251-392-0/krb5kdc/test-user.keytab.
Sep 27 14:54:26 dist-test-slave-xk1g krb5kdc[15732](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790520866, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T14:54:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T14:54:26Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790520684602251-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790520684602251-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 14:54:26.114046 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6936m5/build/debug/bin/kudu
/tmp/dist-test-task6936m5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790520684602251-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790520684602251-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790520684602251-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790520684602251-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34779
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790520684602251-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40871
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:34779
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790520684602251-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790520684602251-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790520684602251-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 14:54:26.220124 15745 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 14:54:26.220362 15745 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 14:54:26.220402 15745 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 14:54:26.224246 15745 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260927 14:54:26.224283 15745 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260927 14:54:26.224298 15745 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260927 14:54:26.224313 15745 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 14:54:26.224329 15745 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 14:54:26.224346 15745 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 14:54:26.224361 15745 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 14:54:26.224376 15745 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 14:54:26.229444 15745 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40871
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790520684602251-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790520684602251-392-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:34779
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790520684602251-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34779
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790520684602251-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15745
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790520684602251-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:15:17 UTC on ccycloud.quasar-ledxfe.root.comops.site
build id 4676134
I20260927 14:54:26.230345 15745 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 14:54:26.231251 15745 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 14:54:26.239586 15751 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 14:54:26.239539 15753 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 14:54:26.239538 15750 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 14:54:26.239912 15745 server_base.cc:1034] running on GCE node
I20260927 14:54:26.240620 15745 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 14:54:26.241783 15745 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 14:54:26.242980 15745 hybrid_clock.cc:648] HybridClock initialized: now 1790520866242950 us; error 44 us; skew 500 ppm
Sep 27 14:54:26 dist-test-slave-xk1g krb5kdc[15732](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790520866, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 14:54:26.245828 15745 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 14:54:26.246750 15745 webserver.cc:460] Webserver started at http://127.0.98.62:44315/ using document root <none> and password file <none>
I20260927 14:54:26.247335 15745 fs_manager.cc:359] Metadata directory not provided
I20260927 14:54:26.247429 15745 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 14:54:26.247666 15745 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 14:54:26.249786 15745 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790520684602251-392-0/minicluster-data/master-0/data/instance:
uuid: "a992827594d14e428ab1ee64535827e5"
format_stamp: "Formatted at 2026-09-27 14:54:26 on dist-test-slave-xk1g"
server_key: "d50aee429206bc58e096762fd7248ed8"
server_key_iv: "82418a00305e96599705060c600e3e83"
server_key_version: "encryptionkey@0"
I20260927 14:54:26.250351 15745 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790520684602251-392-0/minicluster-data/master-0/wal/instance:
uuid: "a992827594d14e428ab1ee64535827e5"
format_stamp: "Formatted at 2026-09-27 14:54:26 on dist-test-slave-xk1g"
server_key: "d50aee429206bc58e096762fd7248ed8"
server_key_iv: "82418a00305e96599705060c600e3e83"
server_key_version: "encryptionkey@0"
I20260927 14:54:26.254024 15745 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260927 14:54:26.256541 15760 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 14:54:26.257778 15745 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 14:54:26.257921 15745 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790520684602251-392-0/minicluster-data/master-0/data,/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790520684602251-392-0/minicluster-data/master-0/wal
uuid: "a992827594d14e428ab1ee64535827e5"
format_stamp: "Formatted at 2026-09-27 14:54:26 on dist-test-slave-xk1g"
server_key: "d50aee429206bc58e096762fd7248ed8"
server_key_iv: "82418a00305e96599705060c600e3e83"
server_key_version: "encryptionkey@0"
I20260927 14:54:26.258033 15745 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790520684602251-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790520684602251-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790520684602251-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 14:54:26.275411 15745 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 14:54:26.278772 15745 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 14:54:26.278986 15745 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 14:54:26.286001 15745 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:34779
I20260927 14:54:26.286294 15812 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:34779 every 8 connection(s)
I20260927 14:54:26.286983 15745 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790520684602251-392-0/minicluster-data/master-0/data/info.pb
I20260927 14:54:26.289801 15813 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 14:54:26.290128 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6936m5/build/debug/bin/kudu as pid 15745
I20260927 14:54:26.290266 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790520684602251-392-0/minicluster-data/master-0/wal/instance
I20260927 14:54:26.290581 392 external_mini_cluster.cc:1428] Setting key ff20c468b82c9672cabc5c05fd0ea4f2
I20260927 14:54:26.296950 15813 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a992827594d14e428ab1ee64535827e5: Bootstrap starting.
I20260927 14:54:26.299484 15813 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a992827594d14e428ab1ee64535827e5: Neither blocks nor log segments found. Creating new log.
Sep 27 14:54:26 dist-test-slave-xk1g krb5kdc[15732](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790520866, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 14:54:26.300273 15813 log.cc:824] T 00000000000000000000000000000000 P a992827594d14e428ab1ee64535827e5: Log is configured to *not* fsync() on all Append() calls
I20260927 14:54:26.302295 15813 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a992827594d14e428ab1ee64535827e5: No bootstrap required, opened a new log
I20260927 14:54:26.305207 15816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:54:26.291953 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51814 (local address 127.0.98.62:34779)
0927 14:54:26.292372 (+ 419us) server_negotiation.cc:206] Beginning negotiation
0927 14:54:26.292390 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0927 14:54:26.292436 (+ 46us) server_negotiation.cc:407] Connection header received
0927 14:54:26.293288 (+ 852us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:54:26.293312 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:54:26.293749 (+ 437us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:54:26.294124 (+ 375us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:54:26.294793 (+ 669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:54:26.296295 (+ 1502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:54:26.297172 (+ 877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:54:26.297857 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:54:26.298017 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:54:26.300543 (+ 2526us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:54:26.300550 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:54:26.300567 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 14:54:26.300599 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:54:26.302367 (+ 1768us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:54:26.302882 (+ 515us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:54:26.302886 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:54:26.302891 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:54:26.302996 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:54:26.303294 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:54:26.303296 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:54:26.303297 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:54:26.303666 (+ 369us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:54:26.303829 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0927 14:54:26.304123 (+ 294us) server_negotiation.cc:299] Negotiation successful
0927 14:54:26.304425 (+ 302us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":271,"thread_start_us":114,"threads_started":1}
I20260927 14:54:26.306392 15813 raft_consensus.cc:348] T 00000000000000000000000000000000 P a992827594d14e428ab1ee64535827e5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a992827594d14e428ab1ee64535827e5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34779 } }
I20260927 14:54:26.306648 15813 raft_consensus.cc:374] T 00000000000000000000000000000000 P a992827594d14e428ab1ee64535827e5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 14:54:26.306720 15813 raft_consensus.cc:729] T 00000000000000000000000000000000 P a992827594d14e428ab1ee64535827e5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a992827594d14e428ab1ee64535827e5, State: Initialized, Role: FOLLOWER
I20260927 14:54:26.307184 15813 consensus_queue.cc:260] T 00000000000000000000000000000000 P a992827594d14e428ab1ee64535827e5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a992827594d14e428ab1ee64535827e5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34779 } }
I20260927 14:54:26.307307 15813 raft_consensus.cc:388] T 00000000000000000000000000000000 P a992827594d14e428ab1ee64535827e5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 14:54:26.307386 15813 raft_consensus.cc:482] T 00000000000000000000000000000000 P a992827594d14e428ab1ee64535827e5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 14:54:26.307528 15813 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a992827594d14e428ab1ee64535827e5 [term 0 FOLLOWER]: Advancing to term 1
I20260927 14:54:26.308455 15813 raft_consensus.cc:504] T 00000000000000000000000000000000 P a992827594d14e428ab1ee64535827e5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a992827594d14e428ab1ee64535827e5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34779 } }
I20260927 14:54:26.308792 15813 leader_election.cc:302] T 00000000000000000000000000000000 P a992827594d14e428ab1ee64535827e5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a992827594d14e428ab1ee64535827e5; no voters:
I20260927 14:54:26.309171 15813 leader_election.cc:288] T 00000000000000000000000000000000 P a992827594d14e428ab1ee64535827e5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 14:54:26.309284 15818 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a992827594d14e428ab1ee64535827e5 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 14:54:26.309548 15818 raft_consensus.cc:686] T 00000000000000000000000000000000 P a992827594d14e428ab1ee64535827e5 [term 1 LEADER]: Becoming Leader. State: Replica: a992827594d14e428ab1ee64535827e5, State: Running, Role: LEADER
I20260927 14:54:26.309913 15818 consensus_queue.cc:237] T 00000000000000000000000000000000 P a992827594d14e428ab1ee64535827e5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a992827594d14e428ab1ee64535827e5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34779 } }
I20260927 14:54:26.310408 15813 sys_catalog.cc:564] T 00000000000000000000000000000000 P a992827594d14e428ab1ee64535827e5 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 14:54:26.311663 15819 sys_catalog.cc:455] T 00000000000000000000000000000000 P a992827594d14e428ab1ee64535827e5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a992827594d14e428ab1ee64535827e5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a992827594d14e428ab1ee64535827e5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34779 } } }
I20260927 14:54:26.311657 15820 sys_catalog.cc:455] T 00000000000000000000000000000000 P a992827594d14e428ab1ee64535827e5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a992827594d14e428ab1ee64535827e5. Latest consensus state: current_term: 1 leader_uuid: "a992827594d14e428ab1ee64535827e5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a992827594d14e428ab1ee64535827e5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34779 } } }
I20260927 14:54:26.311806 15820 sys_catalog.cc:458] T 00000000000000000000000000000000 P a992827594d14e428ab1ee64535827e5 [sys.catalog]: This master's current role is: LEADER
I20260927 14:54:26.311806 15819 sys_catalog.cc:458] T 00000000000000000000000000000000 P a992827594d14e428ab1ee64535827e5 [sys.catalog]: This master's current role is: LEADER
I20260927 14:54:26.312245 15828 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 14:54:26.314919 15828 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 14:54:26.320415 15828 catalog_manager.cc:1347] Generated new cluster ID: d488d1bfd5274fd3ab35889a54cbc5d0
I20260927 14:54:26.320482 15828 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 14:54:26.342265 15828 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 14:54:26.343205 15828 catalog_manager.cc:1504] Loading token signing keys...
I20260927 14:54:26.348956 15828 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a992827594d14e428ab1ee64535827e5: Generated new TSK 0
I20260927 14:54:26.349679 15828 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790520684602251-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790520684602251-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260927 14:54:26.377913 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6936m5/build/debug/bin/kudu
/tmp/dist-test-task6936m5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790520684602251-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790520684602251-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790520684602251-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790520684602251-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34779
--keytab_file=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790520684602251-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40871
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790520684602251-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790520684602251-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790520684602251-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 14:54:26.487439 15839 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 14:54:26.487694 15839 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 14:54:26.487746 15839 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 14:54:26.491699 15839 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 14:54:26.491762 15839 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 14:54:26.491820 15839 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 14:54:26.496949 15839 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40871
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790520684602251-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790520684602251-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790520684602251-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790520684602251-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34779
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15839
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790520684602251-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:15:17 UTC on ccycloud.quasar-ledxfe.root.comops.site
build id 4676134
I20260927 14:54:26.497982 15839 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 14:54:26.498924 15839 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 14:54:26.508033 15847 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 14:54:26.508080 15839 server_base.cc:1034] running on GCE node
W20260927 14:54:26.508177 15844 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 14:54:26.508239 15845 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 14:54:26.508735 15839 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 14:54:26.509418 15839 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 14:54:26.510603 15839 hybrid_clock.cc:648] HybridClock initialized: now 1790520866510566 us; error 44 us; skew 500 ppm
Sep 27 14:54:26 dist-test-slave-xk1g krb5kdc[15732](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790520866, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 14:54:26.513628 15839 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 14:54:26.514537 15839 webserver.cc:460] Webserver started at http://127.0.98.1:40025/ using document root <none> and password file <none>
I20260927 14:54:26.515092 15839 fs_manager.cc:359] Metadata directory not provided
I20260927 14:54:26.515182 15839 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 14:54:26.515412 15839 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 14:54:26.517500 15839 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790520684602251-392-0/minicluster-data/ts-0/data/instance:
uuid: "f8ee67065e464c1b95de6b57be9697ad"
format_stamp: "Formatted at 2026-09-27 14:54:26 on dist-test-slave-xk1g"
server_key: "952ee1f62b1ebc8e55e8c976234b4dce"
server_key_iv: "592e8d010d72b7b41a99653c5bc2d2e6"
server_key_version: "encryptionkey@0"
I20260927 14:54:26.518054 15839 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790520684602251-392-0/minicluster-data/ts-0/wal/instance:
uuid: "f8ee67065e464c1b95de6b57be9697ad"
format_stamp: "Formatted at 2026-09-27 14:54:26 on dist-test-slave-xk1g"
server_key: "952ee1f62b1ebc8e55e8c976234b4dce"
server_key_iv: "592e8d010d72b7b41a99653c5bc2d2e6"
server_key_version: "encryptionkey@0"
I20260927 14:54:26.521739 15839 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260927 14:54:26.524151 15854 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 14:54:26.525429 15839 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260927 14:54:26.525556 15839 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790520684602251-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790520684602251-392-0/minicluster-data/ts-0/wal
uuid: "f8ee67065e464c1b95de6b57be9697ad"
format_stamp: "Formatted at 2026-09-27 14:54:26 on dist-test-slave-xk1g"
server_key: "952ee1f62b1ebc8e55e8c976234b4dce"
server_key_iv: "592e8d010d72b7b41a99653c5bc2d2e6"
server_key_version: "encryptionkey@0"
I20260927 14:54:26.525662 15839 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790520684602251-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790520684602251-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790520684602251-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 14:54:26.534538 15839 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 14:54:26.537508 15839 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 14:54:26.537700 15839 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 14:54:26.538251 15839 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 14:54:26.539012 15839 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 14:54:26.539080 15839 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 14:54:26.539144 15839 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 14:54:26.539192 15839 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 14:54:26.548316 15839 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:34745
I20260927 14:54:26.548346 15967 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:34745 every 8 connection(s)
I20260927 14:54:26.549212 15839 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790520684602251-392-0/minicluster-data/ts-0/data/info.pb
I20260927 14:54:26.555105 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6936m5/build/debug/bin/kudu as pid 15839
I20260927 14:54:26.555227 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790520684602251-392-0/minicluster-data/ts-0/wal/instance
I20260927 14:54:26.555526 392 external_mini_cluster.cc:1428] Setting key bf04cbdc013496a47fc2e35c096167e4
Sep 27 14:54:26 dist-test-slave-xk1g krb5kdc[15732](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790520866, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 14:54:26.563362 15816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:54:26.550946 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:46129 (local address 127.0.98.62:34779)
0927 14:54:26.551085 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0927 14:54:26.551088 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 14:54:26.551981 (+ 893us) server_negotiation.cc:407] Connection header received
0927 14:54:26.553287 (+ 1306us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:54:26.553290 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:54:26.553335 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:54:26.553415 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:54:26.554539 (+ 1124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:54:26.555262 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:54:26.556719 (+ 1457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:54:26.556966 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:54:26.557176 (+ 210us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:54:26.559682 (+ 2506us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:54:26.559686 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:54:26.559689 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 14:54:26.559716 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:54:26.561355 (+ 1639us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:54:26.561851 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:54:26.561855 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:54:26.561857 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:54:26.561946 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:54:26.562295 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:54:26.562302 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:54:26.562307 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:54:26.562489 (+ 182us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:54:26.562596 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0927 14:54:26.563079 (+ 483us) server_negotiation.cc:299] Negotiation successful
0927 14:54:26.563203 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260927 14:54:26.564188 15970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:54:26.551341 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34779 (local address 127.0.98.1:46129)
0927 14:54:26.551821 (+ 480us) negotiation.cc:107] Waiting for socket to connect
0927 14:54:26.551874 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0927 14:54:26.553094 (+ 1220us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:54:26.553583 (+ 489us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:54:26.553589 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:54:26.553940 (+ 351us) client_negotiation.cc:189] Negotiated authn=SASL
0927 14:54:26.554404 (+ 464us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:54:26.554416 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:54:26.555419 (+ 1003us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:54:26.555425 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:54:26.556584 (+ 1159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:54:26.556594 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:54:26.557209 (+ 615us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:54:26.557213 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:54:26.557388 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:54:26.557983 (+ 595us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 14:54:26.557999 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 14:54:26.559525 (+ 1526us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 14:54:26.561469 (+ 1944us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 14:54:26.561475 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 14:54:26.561489 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0927 14:54:26.561743 (+ 254us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 14:54:26.562052 (+ 309us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 14:54:26.562055 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 14:54:26.562057 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 14:54:26.562200 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 14:54:26.562595 (+ 395us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 14:54:26.562602 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 14:54:26.562844 (+ 242us) client_negotiation.cc:769] Sending connection context
0927 14:54:26.563027 (+ 183us) client_negotiation.cc:240] Negotiation successful
0927 14:54:26.563366 (+ 339us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":267,"thread_start_us":132,"threads_started":1}
I20260927 14:54:26.565436 15968 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34779
I20260927 14:54:26.565653 15968 heartbeater.cc:464] Registering TS with master...
I20260927 14:54:26.566124 15968 heartbeater.cc:511] Master 127.0.98.62:34779 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260927 14:54:26.567752 15774 ts_manager.cc:194] Registered new tserver with Master: f8ee67065e464c1b95de6b57be9697ad (127.0.98.1:34745)
I20260927 14:54:26.569022 15774 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:46129
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790520684602251-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790520684602251-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260927 14:54:26.581923 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6936m5/build/debug/bin/kudu
/tmp/dist-test-task6936m5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790520684602251-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790520684602251-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790520684602251-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790520684602251-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34779
--keytab_file=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790520684602251-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40871
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790520684602251-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790520684602251-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790520684602251-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 14:54:26.690248 15973 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 14:54:26.690492 15973 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 14:54:26.690537 15973 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 14:54:26.694403 15973 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 14:54:26.694453 15973 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 14:54:26.694509 15973 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 14:54:26.699493 15973 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40871
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790520684602251-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790520684602251-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790520684602251-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790520684602251-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34779
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15973
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790520684602251-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:15:17 UTC on ccycloud.quasar-ledxfe.root.comops.site
build id 4676134
I20260927 14:54:26.700403 15973 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 14:54:26.701418 15973 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 14:54:26.710665 15979 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 14:54:26.710642 15978 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 14:54:26.710667 15981 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 14:54:26.710876 15973 server_base.cc:1034] running on GCE node
I20260927 14:54:26.711376 15973 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 14:54:26.712003 15973 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 14:54:26.713189 15973 hybrid_clock.cc:648] HybridClock initialized: now 1790520866713174 us; error 28 us; skew 500 ppm
Sep 27 14:54:26 dist-test-slave-xk1g krb5kdc[15732](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790520866, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 14:54:26.715917 15973 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 14:54:26.716707 15973 webserver.cc:460] Webserver started at http://127.0.98.2:33621/ using document root <none> and password file <none>
I20260927 14:54:26.717316 15973 fs_manager.cc:359] Metadata directory not provided
I20260927 14:54:26.717404 15973 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 14:54:26.717635 15973 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 14:54:26.719563 15973 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790520684602251-392-0/minicluster-data/ts-1/data/instance:
uuid: "f3e6feeb2c794c849ab9ef7fbef5fb4c"
format_stamp: "Formatted at 2026-09-27 14:54:26 on dist-test-slave-xk1g"
server_key: "d4aafd3368d18eac59a1a1b0b80dca96"
server_key_iv: "bd37185f913e046786f09f6fba5e5470"
server_key_version: "encryptionkey@0"
I20260927 14:54:26.720120 15973 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790520684602251-392-0/minicluster-data/ts-1/wal/instance:
uuid: "f3e6feeb2c794c849ab9ef7fbef5fb4c"
format_stamp: "Formatted at 2026-09-27 14:54:26 on dist-test-slave-xk1g"
server_key: "d4aafd3368d18eac59a1a1b0b80dca96"
server_key_iv: "bd37185f913e046786f09f6fba5e5470"
server_key_version: "encryptionkey@0"
I20260927 14:54:26.723735 15973 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260927 14:54:26.726071 15988 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 14:54:26.727233 15973 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 14:54:26.727368 15973 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790520684602251-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790520684602251-392-0/minicluster-data/ts-1/wal
uuid: "f3e6feeb2c794c849ab9ef7fbef5fb4c"
format_stamp: "Formatted at 2026-09-27 14:54:26 on dist-test-slave-xk1g"
server_key: "d4aafd3368d18eac59a1a1b0b80dca96"
server_key_iv: "bd37185f913e046786f09f6fba5e5470"
server_key_version: "encryptionkey@0"
I20260927 14:54:26.727478 15973 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790520684602251-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790520684602251-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790520684602251-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 14:54:26.737205 15973 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 14:54:26.740515 15973 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 14:54:26.740731 15973 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 14:54:26.741382 15973 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 14:54:26.742235 15973 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 14:54:26.742307 15973 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 14:54:26.742380 15973 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 14:54:26.742424 15973 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 14:54:26.751327 15973 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:37931
I20260927 14:54:26.751353 16101 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:37931 every 8 connection(s)
I20260927 14:54:26.752228 15973 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790520684602251-392-0/minicluster-data/ts-1/data/info.pb
I20260927 14:54:26.757544 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6936m5/build/debug/bin/kudu as pid 15973
I20260927 14:54:26.757687 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790520684602251-392-0/minicluster-data/ts-1/wal/instance
I20260927 14:54:26.757972 392 external_mini_cluster.cc:1428] Setting key fe80d71942fba486738b8b9a9227e0bc
Sep 27 14:54:26 dist-test-slave-xk1g krb5kdc[15732](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790520866, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 14:54:26.766999 15816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:54:26.754330 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:38885 (local address 127.0.98.62:34779)
0927 14:54:26.754490 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0927 14:54:26.754494 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 14:54:26.755277 (+ 783us) server_negotiation.cc:407] Connection header received
0927 14:54:26.756543 (+ 1266us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:54:26.756546 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:54:26.756614 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:54:26.756703 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:54:26.758101 (+ 1398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:54:26.758820 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:54:26.760180 (+ 1360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:54:26.760579 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:54:26.760716 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:54:26.763436 (+ 2720us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:54:26.763440 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:54:26.763443 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 14:54:26.763468 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:54:26.764915 (+ 1447us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:54:26.765467 (+ 552us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:54:26.765473 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:54:26.765476 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:54:26.765563 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:54:26.765908 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:54:26.765911 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:54:26.765913 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:54:26.766093 (+ 180us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:54:26.766200 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0927 14:54:26.766701 (+ 501us) server_negotiation.cc:299] Negotiation successful
0927 14:54:26.766830 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260927 14:54:26.767844 16104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:54:26.754678 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34779 (local address 127.0.98.2:38885)
0927 14:54:26.755113 (+ 435us) negotiation.cc:107] Waiting for socket to connect
0927 14:54:26.755154 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0927 14:54:26.756307 (+ 1153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:54:26.756877 (+ 570us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:54:26.756883 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:54:26.757339 (+ 456us) client_negotiation.cc:189] Negotiated authn=SASL
0927 14:54:26.757939 (+ 600us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:54:26.757957 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:54:26.758945 (+ 988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:54:26.758948 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:54:26.760045 (+ 1097us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:54:26.760055 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:54:26.760738 (+ 683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:54:26.760752 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:54:26.760964 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:54:26.761729 (+ 765us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 14:54:26.761747 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0927 14:54:26.763271 (+ 1524us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 14:54:26.765036 (+ 1765us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 14:54:26.765042 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 14:54:26.765080 (+ 38us) client_negotiation.cc:760] calling sasl_client_step()
0927 14:54:26.765358 (+ 278us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 14:54:26.765659 (+ 301us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 14:54:26.765663 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 14:54:26.765665 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 14:54:26.765806 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 14:54:26.766200 (+ 394us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 14:54:26.766207 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 14:54:26.766461 (+ 254us) client_negotiation.cc:769] Sending connection context
0927 14:54:26.766645 (+ 184us) client_negotiation.cc:240] Negotiation successful
0927 14:54:26.766965 (+ 320us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":256,"thread_start_us":121,"threads_started":1}
I20260927 14:54:26.768983 16102 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34779
I20260927 14:54:26.769258 16102 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260927 14:54:26.769754 16102 heartbeater.cc:511] Master 127.0.98.62:34779 requested a full tablet report, sending...
I20260927 14:54:26.772034 15774 ts_manager.cc:194] Registered new tserver with Master: f3e6feeb2c794c849ab9ef7fbef5fb4c (127.0.98.2:37931)
I20260927 14:54:26.772733 15774 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:38885
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790520684602251-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790520684602251-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260927 14:54:26.785614 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6936m5/build/debug/bin/kudu
/tmp/dist-test-task6936m5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790520684602251-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790520684602251-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790520684602251-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790520684602251-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34779
--keytab_file=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790520684602251-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40871
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790520684602251-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790520684602251-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790520684602251-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 14:54:26.895996 16107 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 14:54:26.896314 16107 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 14:54:26.896389 16107 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 14:54:26.900416 16107 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 14:54:26.900511 16107 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 14:54:26.900607 16107 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 14:54:26.905921 16107 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40871
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790520684602251-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790520684602251-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790520684602251-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790520684602251-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34779
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16107
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790520684602251-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:15:17 UTC on ccycloud.quasar-ledxfe.root.comops.site
build id 4676134
I20260927 14:54:26.906996 16107 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 14:54:26.908015 16107 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 14:54:26.917734 16115 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 14:54:26.917783 16113 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 14:54:26.917734 16112 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 14:54:26.918045 16107 server_base.cc:1034] running on GCE node
I20260927 14:54:26.918432 16107 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 14:54:26.919073 16107 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 14:54:26.920291 16107 hybrid_clock.cc:648] HybridClock initialized: now 1790520866920253 us; error 48 us; skew 500 ppm
Sep 27 14:54:26 dist-test-slave-xk1g krb5kdc[15732](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790520866, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 14:54:26.923039 16107 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 14:54:26.923954 16107 webserver.cc:460] Webserver started at http://127.0.98.3:45735/ using document root <none> and password file <none>
I20260927 14:54:26.924515 16107 fs_manager.cc:359] Metadata directory not provided
I20260927 14:54:26.924628 16107 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 14:54:26.924850 16107 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 14:54:26.926968 16107 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790520684602251-392-0/minicluster-data/ts-2/data/instance:
uuid: "9eab78e4338043749a6dbec46092f279"
format_stamp: "Formatted at 2026-09-27 14:54:26 on dist-test-slave-xk1g"
server_key: "678491584feb82d497710ade28548c61"
server_key_iv: "acf9c8bba7ec8bb4f65321f6ec1f109c"
server_key_version: "encryptionkey@0"
I20260927 14:54:26.927534 16107 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790520684602251-392-0/minicluster-data/ts-2/wal/instance:
uuid: "9eab78e4338043749a6dbec46092f279"
format_stamp: "Formatted at 2026-09-27 14:54:26 on dist-test-slave-xk1g"
server_key: "678491584feb82d497710ade28548c61"
server_key_iv: "acf9c8bba7ec8bb4f65321f6ec1f109c"
server_key_version: "encryptionkey@0"
I20260927 14:54:26.931433 16107 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260927 14:54:26.933986 16122 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 14:54:26.935213 16107 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.003s
I20260927 14:54:26.935444 16107 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790520684602251-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790520684602251-392-0/minicluster-data/ts-2/wal
uuid: "9eab78e4338043749a6dbec46092f279"
format_stamp: "Formatted at 2026-09-27 14:54:26 on dist-test-slave-xk1g"
server_key: "678491584feb82d497710ade28548c61"
server_key_iv: "acf9c8bba7ec8bb4f65321f6ec1f109c"
server_key_version: "encryptionkey@0"
I20260927 14:54:26.935566 16107 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790520684602251-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790520684602251-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790520684602251-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 14:54:26.977941 16107 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 14:54:26.981390 16107 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 14:54:26.981621 16107 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 14:54:26.982175 16107 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 14:54:26.982939 16107 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 14:54:26.982983 16107 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 14:54:26.983021 16107 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 14:54:26.983037 16107 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 14:54:26.991757 16107 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:46285
I20260927 14:54:26.991778 16235 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:46285 every 8 connection(s)
I20260927 14:54:26.992674 16107 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790520684602251-392-0/minicluster-data/ts-2/data/info.pb
I20260927 14:54:27.002024 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6936m5/build/debug/bin/kudu as pid 16107
I20260927 14:54:27.002156 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790520684602251-392-0/minicluster-data/ts-2/wal/instance
Sep 27 14:54:27 dist-test-slave-xk1g krb5kdc[15732](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790520866, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 14:54:27.002471 392 external_mini_cluster.cc:1428] Setting key 4daebb7265c1a8febd5b20f4027ea64b
I20260927 14:54:27.007071 15816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:54:26.994530 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:41695 (local address 127.0.98.62:34779)
0927 14:54:26.994681 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0927 14:54:26.994684 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 14:54:26.995519 (+ 835us) server_negotiation.cc:407] Connection header received
0927 14:54:26.996845 (+ 1326us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:54:26.996848 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:54:26.996927 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:54:26.997028 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:54:26.998239 (+ 1211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:54:26.998776 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:54:26.999933 (+ 1157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:54:27.000180 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:54:27.000304 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:54:27.003153 (+ 2849us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:54:27.003157 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:54:27.003160 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 14:54:27.003195 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:54:27.004969 (+ 1774us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:54:27.005562 (+ 593us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:54:27.005564 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:54:27.005566 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:54:27.005639 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:54:27.006029 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:54:27.006032 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:54:27.006033 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:54:27.006233 (+ 200us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:54:27.006317 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0927 14:54:27.006849 (+ 532us) server_negotiation.cc:299] Negotiation successful
0927 14:54:27.006950 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260927 14:54:27.007946 16238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:54:26.994912 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34779 (local address 127.0.98.3:41695)
0927 14:54:26.995363 (+ 451us) negotiation.cc:107] Waiting for socket to connect
0927 14:54:26.995406 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0927 14:54:26.996623 (+ 1217us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:54:26.997243 (+ 620us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:54:26.997250 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:54:26.997614 (+ 364us) client_negotiation.cc:189] Negotiated authn=SASL
0927 14:54:26.998094 (+ 480us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:54:26.998107 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:54:26.998905 (+ 798us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:54:26.998909 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:54:26.999806 (+ 897us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:54:26.999815 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:54:27.000338 (+ 523us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:54:27.000343 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:54:27.000517 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:54:27.001167 (+ 650us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 14:54:27.001184 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 14:54:27.002975 (+ 1791us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 14:54:27.005171 (+ 2196us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 14:54:27.005176 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 14:54:27.005190 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0927 14:54:27.005465 (+ 275us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 14:54:27.005724 (+ 259us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 14:54:27.005726 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 14:54:27.005727 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 14:54:27.005870 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 14:54:27.006361 (+ 491us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 14:54:27.006366 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 14:54:27.006624 (+ 258us) client_negotiation.cc:769] Sending connection context
0927 14:54:27.006785 (+ 161us) client_negotiation.cc:240] Negotiation successful
0927 14:54:27.007066 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":264,"thread_start_us":118,"threads_started":1}
I20260927 14:54:27.009162 16236 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34779
I20260927 14:54:27.009423 16236 heartbeater.cc:464] Registering TS with master...
I20260927 14:54:27.009920 16236 heartbeater.cc:511] Master 127.0.98.62:34779 requested a full tablet report, sending...
I20260927 14:54:27.011078 15774 ts_manager.cc:194] Registered new tserver with Master: 9eab78e4338043749a6dbec46092f279 (127.0.98.3:46285)
I20260927 14:54:27.011678 15774 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:41695
I20260927 14:54:27.016537 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 14:54:27.026235 15816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:54:27.018542 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51820 (local address 127.0.98.62:34779)
0927 14:54:27.018707 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0927 14:54:27.018711 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 14:54:27.018909 (+ 198us) server_negotiation.cc:407] Connection header received
0927 14:54:27.019013 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:54:27.019015 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:54:27.019057 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:54:27.019127 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:54:27.019849 (+ 722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:54:27.020367 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:54:27.021318 (+ 951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:54:27.021534 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:54:27.021631 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:54:27.022981 (+ 1350us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:54:27.022985 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:54:27.022987 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 14:54:27.023012 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:54:27.024594 (+ 1582us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:54:27.024966 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:54:27.024968 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:54:27.024970 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:54:27.025041 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:54:27.025345 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:54:27.025348 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:54:27.025349 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:54:27.025574 (+ 225us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:54:27.025709 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0927 14:54:27.025907 (+ 198us) server_negotiation.cc:299] Negotiation successful
0927 14:54:27.026003 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260927 14:54:27.028959 15774 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51820:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 14:54:27.032210 15774 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 14:54:27.046343 16250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:54:27.040160 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:37931 (local address 127.0.0.1:54160)
0927 14:54:27.041330 (+ 1170us) negotiation.cc:107] Waiting for socket to connect
0927 14:54:27.041344 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 14:54:27.041428 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:54:27.041940 (+ 512us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:54:27.041947 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:54:27.041972 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 14:54:27.042097 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:54:27.042104 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:54:27.043344 (+ 1240us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:54:27.043349 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:54:27.044774 (+ 1425us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:54:27.044784 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:54:27.045798 (+ 1014us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:54:27.045802 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:54:27.045908 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:54:27.045951 (+ 43us) client_negotiation.cc:769] Sending connection context
0927 14:54:27.046058 (+ 107us) client_negotiation.cc:240] Negotiation successful
0927 14:54:27.046187 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1107,"mutex_wait_us":9,"thread_start_us":101,"threads_started":1}
I20260927 14:54:27.046383 16251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:54:27.040882 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54160 (local address 127.0.98.2:37931)
0927 14:54:27.041428 (+ 546us) server_negotiation.cc:206] Beginning negotiation
0927 14:54:27.041440 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 14:54:27.041464 (+ 24us) server_negotiation.cc:407] Connection header received
0927 14:54:27.041542 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:54:27.041548 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:54:27.041761 (+ 213us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:54:27.041924 (+ 163us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 14:54:27.042222 (+ 298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:54:27.043222 (+ 1000us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:54:27.044878 (+ 1656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:54:27.045669 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:54:27.045788 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:54:27.045931 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0927 14:54:27.046108 (+ 177us) server_negotiation.cc:299] Negotiation successful
0927 14:54:27.046208 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":436,"thread_start_us":95,"threads_started":1}
I20260927 14:54:27.046691 16249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:54:27.040263 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60798 (local address 127.0.98.3:46285)
0927 14:54:27.042121 (+ 1858us) server_negotiation.cc:206] Beginning negotiation
0927 14:54:27.042137 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0927 14:54:27.042154 (+ 17us) server_negotiation.cc:407] Connection header received
0927 14:54:27.042245 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:54:27.042250 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:54:27.042430 (+ 180us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:54:27.042568 (+ 138us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 14:54:27.042963 (+ 395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:54:27.044069 (+ 1106us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:54:27.045323 (+ 1254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:54:27.045950 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:54:27.046069 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:54:27.046199 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0927 14:54:27.046432 (+ 233us) server_negotiation.cc:299] Negotiation successful
0927 14:54:27.046534 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1758,"thread_start_us":153,"threads_started":1}
I20260927 14:54:27.047024 16252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:54:27.040880 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35870 (local address 127.0.98.1:34745)
0927 14:54:27.041654 (+ 774us) server_negotiation.cc:206] Beginning negotiation
0927 14:54:27.041667 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 14:54:27.041902 (+ 235us) server_negotiation.cc:407] Connection header received
0927 14:54:27.041980 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:54:27.041985 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:54:27.042183 (+ 198us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:54:27.042348 (+ 165us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 14:54:27.042564 (+ 216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:54:27.043604 (+ 1040us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:54:27.044815 (+ 1211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:54:27.045669 (+ 854us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:54:27.045788 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:54:27.045931 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0927 14:54:27.046768 (+ 837us) server_negotiation.cc:299] Negotiation successful
0927 14:54:27.046872 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":672,"thread_start_us":115,"threads_started":1}
I20260927 14:54:27.047025 16247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:54:27.040086 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:46285 (local address 127.0.0.1:60798)
0927 14:54:27.040890 (+ 804us) negotiation.cc:107] Waiting for socket to connect
0927 14:54:27.040939 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0927 14:54:27.041093 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:54:27.042728 (+ 1635us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:54:27.042731 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:54:27.042747 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 14:54:27.042825 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:54:27.042830 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:54:27.044185 (+ 1355us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:54:27.044188 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:54:27.045210 (+ 1022us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:54:27.045218 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:54:27.046094 (+ 876us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:54:27.046098 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:54:27.046186 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:54:27.046207 (+ 21us) client_negotiation.cc:769] Sending connection context
0927 14:54:27.046283 (+ 76us) client_negotiation.cc:240] Negotiation successful
0927 14:54:27.046350 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":622,"thread_start_us":83,"threads_started":1}
I20260927 14:54:27.046674 16248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:54:27.040438 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:34745 (local address 127.0.0.1:35870)
0927 14:54:27.041782 (+ 1344us) negotiation.cc:107] Waiting for socket to connect
0927 14:54:27.041795 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 14:54:27.041891 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:54:27.042326 (+ 435us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:54:27.042329 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:54:27.042346 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 14:54:27.042447 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:54:27.042454 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:54:27.043763 (+ 1309us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:54:27.043767 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:54:27.044682 (+ 915us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:54:27.044691 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:54:27.046271 (+ 1580us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:54:27.046275 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:54:27.046360 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:54:27.046393 (+ 33us) client_negotiation.cc:769] Sending connection context
0927 14:54:27.046470 (+ 77us) client_negotiation.cc:240] Negotiation successful
0927 14:54:27.046551 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1287,"thread_start_us":83,"threads_started":1}
I20260927 14:54:27.049748 16170 tablet_service.cc:1467] Processing CreateTablet for tablet 3c2df70ba6e3416d9d610abd0756e6f9 (DEFAULT_TABLE table=test-table [id=3866d04318874c01ae4d89b5f00c30a2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 14:54:27.049824 15902 tablet_service.cc:1467] Processing CreateTablet for tablet 3c2df70ba6e3416d9d610abd0756e6f9 (DEFAULT_TABLE table=test-table [id=3866d04318874c01ae4d89b5f00c30a2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 14:54:27.049824 16036 tablet_service.cc:1467] Processing CreateTablet for tablet 3c2df70ba6e3416d9d610abd0756e6f9 (DEFAULT_TABLE table=test-table [id=3866d04318874c01ae4d89b5f00c30a2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 14:54:27.050805 16170 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3c2df70ba6e3416d9d610abd0756e6f9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 14:54:27.050807 15902 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3c2df70ba6e3416d9d610abd0756e6f9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 14:54:27.050807 16036 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3c2df70ba6e3416d9d610abd0756e6f9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 14:54:27.056610 16253 tablet_bootstrap.cc:492] T 3c2df70ba6e3416d9d610abd0756e6f9 P 9eab78e4338043749a6dbec46092f279: Bootstrap starting.
I20260927 14:54:27.058106 16254 tablet_bootstrap.cc:492] T 3c2df70ba6e3416d9d610abd0756e6f9 P f8ee67065e464c1b95de6b57be9697ad: Bootstrap starting.
I20260927 14:54:27.058472 16255 tablet_bootstrap.cc:492] T 3c2df70ba6e3416d9d610abd0756e6f9 P f3e6feeb2c794c849ab9ef7fbef5fb4c: Bootstrap starting.
I20260927 14:54:27.058883 16253 tablet_bootstrap.cc:654] T 3c2df70ba6e3416d9d610abd0756e6f9 P 9eab78e4338043749a6dbec46092f279: Neither blocks nor log segments found. Creating new log.
I20260927 14:54:27.059947 16253 log.cc:824] T 3c2df70ba6e3416d9d610abd0756e6f9 P 9eab78e4338043749a6dbec46092f279: Log is configured to *not* fsync() on all Append() calls
I20260927 14:54:27.060787 16255 tablet_bootstrap.cc:654] T 3c2df70ba6e3416d9d610abd0756e6f9 P f3e6feeb2c794c849ab9ef7fbef5fb4c: Neither blocks nor log segments found. Creating new log.
I20260927 14:54:27.061669 16255 log.cc:824] T 3c2df70ba6e3416d9d610abd0756e6f9 P f3e6feeb2c794c849ab9ef7fbef5fb4c: Log is configured to *not* fsync() on all Append() calls
I20260927 14:54:27.062199 16253 tablet_bootstrap.cc:492] T 3c2df70ba6e3416d9d610abd0756e6f9 P 9eab78e4338043749a6dbec46092f279: No bootstrap required, opened a new log
I20260927 14:54:27.062242 16254 tablet_bootstrap.cc:654] T 3c2df70ba6e3416d9d610abd0756e6f9 P f8ee67065e464c1b95de6b57be9697ad: Neither blocks nor log segments found. Creating new log.
I20260927 14:54:27.062422 16253 ts_tablet_manager.cc:1397] T 3c2df70ba6e3416d9d610abd0756e6f9 P 9eab78e4338043749a6dbec46092f279: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260927 14:54:27.063230 16254 log.cc:824] T 3c2df70ba6e3416d9d610abd0756e6f9 P f8ee67065e464c1b95de6b57be9697ad: Log is configured to *not* fsync() on all Append() calls
I20260927 14:54:27.063700 16255 tablet_bootstrap.cc:492] T 3c2df70ba6e3416d9d610abd0756e6f9 P f3e6feeb2c794c849ab9ef7fbef5fb4c: No bootstrap required, opened a new log
I20260927 14:54:27.063970 16255 ts_tablet_manager.cc:1397] T 3c2df70ba6e3416d9d610abd0756e6f9 P f3e6feeb2c794c849ab9ef7fbef5fb4c: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260927 14:54:27.064807 16254 tablet_bootstrap.cc:492] T 3c2df70ba6e3416d9d610abd0756e6f9 P f8ee67065e464c1b95de6b57be9697ad: No bootstrap required, opened a new log
I20260927 14:54:27.065023 16254 ts_tablet_manager.cc:1397] T 3c2df70ba6e3416d9d610abd0756e6f9 P f8ee67065e464c1b95de6b57be9697ad: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260927 14:54:27.066272 16253 raft_consensus.cc:348] T 3c2df70ba6e3416d9d610abd0756e6f9 P 9eab78e4338043749a6dbec46092f279 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9eab78e4338043749a6dbec46092f279" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46285 } } peers { permanent_uuid: "f3e6feeb2c794c849ab9ef7fbef5fb4c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37931 } } peers { permanent_uuid: "f8ee67065e464c1b95de6b57be9697ad" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34745 } }
I20260927 14:54:27.066548 16253 raft_consensus.cc:374] T 3c2df70ba6e3416d9d610abd0756e6f9 P 9eab78e4338043749a6dbec46092f279 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 14:54:27.066633 16253 raft_consensus.cc:729] T 3c2df70ba6e3416d9d610abd0756e6f9 P 9eab78e4338043749a6dbec46092f279 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9eab78e4338043749a6dbec46092f279, State: Initialized, Role: FOLLOWER
I20260927 14:54:27.067082 16253 consensus_queue.cc:260] T 3c2df70ba6e3416d9d610abd0756e6f9 P 9eab78e4338043749a6dbec46092f279 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9eab78e4338043749a6dbec46092f279" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46285 } } peers { permanent_uuid: "f3e6feeb2c794c849ab9ef7fbef5fb4c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37931 } } peers { permanent_uuid: "f8ee67065e464c1b95de6b57be9697ad" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34745 } }
I20260927 14:54:27.067682 16255 raft_consensus.cc:348] T 3c2df70ba6e3416d9d610abd0756e6f9 P f3e6feeb2c794c849ab9ef7fbef5fb4c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9eab78e4338043749a6dbec46092f279" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46285 } } peers { permanent_uuid: "f3e6feeb2c794c849ab9ef7fbef5fb4c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37931 } } peers { permanent_uuid: "f8ee67065e464c1b95de6b57be9697ad" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34745 } }
I20260927 14:54:27.067996 16255 raft_consensus.cc:374] T 3c2df70ba6e3416d9d610abd0756e6f9 P f3e6feeb2c794c849ab9ef7fbef5fb4c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 14:54:27.068065 16253 ts_tablet_manager.cc:1428] T 3c2df70ba6e3416d9d610abd0756e6f9 P 9eab78e4338043749a6dbec46092f279: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260927 14:54:27.068087 16255 raft_consensus.cc:729] T 3c2df70ba6e3416d9d610abd0756e6f9 P f3e6feeb2c794c849ab9ef7fbef5fb4c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f3e6feeb2c794c849ab9ef7fbef5fb4c, State: Initialized, Role: FOLLOWER
I20260927 14:54:27.068166 16254 raft_consensus.cc:348] T 3c2df70ba6e3416d9d610abd0756e6f9 P f8ee67065e464c1b95de6b57be9697ad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9eab78e4338043749a6dbec46092f279" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46285 } } peers { permanent_uuid: "f3e6feeb2c794c849ab9ef7fbef5fb4c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37931 } } peers { permanent_uuid: "f8ee67065e464c1b95de6b57be9697ad" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34745 } }
I20260927 14:54:27.068446 16254 raft_consensus.cc:374] T 3c2df70ba6e3416d9d610abd0756e6f9 P f8ee67065e464c1b95de6b57be9697ad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 14:54:27.068482 16236 heartbeater.cc:503] Master 127.0.98.62:34779 was elected leader, sending a full tablet report...
I20260927 14:54:27.068514 16254 raft_consensus.cc:729] T 3c2df70ba6e3416d9d610abd0756e6f9 P f8ee67065e464c1b95de6b57be9697ad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f8ee67065e464c1b95de6b57be9697ad, State: Initialized, Role: FOLLOWER
I20260927 14:54:27.068599 16255 consensus_queue.cc:260] T 3c2df70ba6e3416d9d610abd0756e6f9 P f3e6feeb2c794c849ab9ef7fbef5fb4c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9eab78e4338043749a6dbec46092f279" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46285 } } peers { permanent_uuid: "f3e6feeb2c794c849ab9ef7fbef5fb4c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37931 } } peers { permanent_uuid: "f8ee67065e464c1b95de6b57be9697ad" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34745 } }
I20260927 14:54:27.068956 16254 consensus_queue.cc:260] T 3c2df70ba6e3416d9d610abd0756e6f9 P f8ee67065e464c1b95de6b57be9697ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9eab78e4338043749a6dbec46092f279" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46285 } } peers { permanent_uuid: "f3e6feeb2c794c849ab9ef7fbef5fb4c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37931 } } peers { permanent_uuid: "f8ee67065e464c1b95de6b57be9697ad" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34745 } }
I20260927 14:54:27.069252 16102 heartbeater.cc:503] Master 127.0.98.62:34779 was elected leader, sending a full tablet report...
I20260927 14:54:27.069557 16255 ts_tablet_manager.cc:1428] T 3c2df70ba6e3416d9d610abd0756e6f9 P f3e6feeb2c794c849ab9ef7fbef5fb4c: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260927 14:54:27.069916 16254 ts_tablet_manager.cc:1428] T 3c2df70ba6e3416d9d610abd0756e6f9 P f8ee67065e464c1b95de6b57be9697ad: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20260927 14:54:27.070199 15968 heartbeater.cc:503] Master 127.0.98.62:34779 was elected leader, sending a full tablet report...
I20260927 14:54:27.092203 16261 raft_consensus.cc:482] T 3c2df70ba6e3416d9d610abd0756e6f9 P f8ee67065e464c1b95de6b57be9697ad [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 14:54:27.092396 16261 raft_consensus.cc:504] T 3c2df70ba6e3416d9d610abd0756e6f9 P f8ee67065e464c1b95de6b57be9697ad [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9eab78e4338043749a6dbec46092f279" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46285 } } peers { permanent_uuid: "f3e6feeb2c794c849ab9ef7fbef5fb4c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37931 } } peers { permanent_uuid: "f8ee67065e464c1b95de6b57be9697ad" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34745 } }
I20260927 14:54:27.093443 16261 leader_election.cc:288] T 3c2df70ba6e3416d9d610abd0756e6f9 P f8ee67065e464c1b95de6b57be9697ad [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9eab78e4338043749a6dbec46092f279 (127.0.98.3:46285), f3e6feeb2c794c849ab9ef7fbef5fb4c (127.0.98.2:37931)
I20260927 14:54:27.097482 16251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:54:27.093908 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:48895 (local address 127.0.98.2:37931)
0927 14:54:27.094020 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0927 14:54:27.094024 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 14:54:27.094180 (+ 156us) server_negotiation.cc:407] Connection header received
0927 14:54:27.094348 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:54:27.094353 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:54:27.094406 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:54:27.094475 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 14:54:27.094826 (+ 351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:54:27.095351 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:54:27.096463 (+ 1112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:54:27.096873 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:54:27.096978 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:54:27.097044 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0927 14:54:27.097290 (+ 246us) server_negotiation.cc:299] Negotiation successful
0927 14:54:27.097358 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260927 14:54:27.097501 16249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:54:27.093806 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:56251 (local address 127.0.98.3:46285)
0927 14:54:27.093951 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0927 14:54:27.093957 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 14:54:27.094288 (+ 331us) server_negotiation.cc:407] Connection header received
0927 14:54:27.094415 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:54:27.094417 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:54:27.094481 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:54:27.094556 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 14:54:27.094860 (+ 304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:54:27.095384 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:54:27.096503 (+ 1119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:54:27.096889 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:54:27.097015 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:54:27.097112 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0927 14:54:27.097316 (+ 204us) server_negotiation.cc:299] Negotiation successful
0927 14:54:27.097389 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260927 14:54:27.097514 16262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:54:27.093692 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:46285 (local address 127.0.98.1:56251)
0927 14:54:27.094158 (+ 466us) negotiation.cc:107] Waiting for socket to connect
0927 14:54:27.094173 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 14:54:27.094312 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:54:27.094585 (+ 273us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:54:27.094588 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:54:27.094602 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 14:54:27.094727 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:54:27.094731 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:54:27.095524 (+ 793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:54:27.095527 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:54:27.096349 (+ 822us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:54:27.096358 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:54:27.097036 (+ 678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:54:27.097040 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:54:27.097173 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:54:27.097191 (+ 18us) client_negotiation.cc:769] Sending connection context
0927 14:54:27.097287 (+ 96us) client_negotiation.cc:240] Negotiation successful
0927 14:54:27.097357 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":393,"mutex_wait_us":157,"thread_start_us":89,"threads_started":1}
I20260927 14:54:27.097520 16263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:54:27.093692 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:37931 (local address 127.0.98.1:48895)
0927 14:54:27.094079 (+ 387us) negotiation.cc:107] Waiting for socket to connect
0927 14:54:27.094094 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 14:54:27.094194 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:54:27.094511 (+ 317us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:54:27.094516 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:54:27.094535 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 14:54:27.094714 (+ 179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:54:27.094720 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:54:27.095463 (+ 743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:54:27.095468 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:54:27.096339 (+ 871us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:54:27.096348 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:54:27.097013 (+ 665us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:54:27.097016 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:54:27.097141 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:54:27.097172 (+ 31us) client_negotiation.cc:769] Sending connection context
0927 14:54:27.097238 (+ 66us) client_negotiation.cc:240] Negotiation successful
0927 14:54:27.097333 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":316,"mutex_wait_us":29,"thread_start_us":108,"threads_started":1}
I20260927 14:54:27.098158 16190 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c2df70ba6e3416d9d610abd0756e6f9" candidate_uuid: "f8ee67065e464c1b95de6b57be9697ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9eab78e4338043749a6dbec46092f279" is_pre_election: true
I20260927 14:54:27.098379 16056 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c2df70ba6e3416d9d610abd0756e6f9" candidate_uuid: "f8ee67065e464c1b95de6b57be9697ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f3e6feeb2c794c849ab9ef7fbef5fb4c" is_pre_election: true
I20260927 14:54:27.098486 16190 raft_consensus.cc:2454] T 3c2df70ba6e3416d9d610abd0756e6f9 P 9eab78e4338043749a6dbec46092f279 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f8ee67065e464c1b95de6b57be9697ad in term 0.
I20260927 14:54:27.098640 16056 raft_consensus.cc:2454] T 3c2df70ba6e3416d9d610abd0756e6f9 P f3e6feeb2c794c849ab9ef7fbef5fb4c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f8ee67065e464c1b95de6b57be9697ad in term 0.
I20260927 14:54:27.099014 15857 leader_election.cc:302] T 3c2df70ba6e3416d9d610abd0756e6f9 P f8ee67065e464c1b95de6b57be9697ad [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 9eab78e4338043749a6dbec46092f279, f3e6feeb2c794c849ab9ef7fbef5fb4c, f8ee67065e464c1b95de6b57be9697ad; no voters:
I20260927 14:54:27.099301 16261 raft_consensus.cc:2781] T 3c2df70ba6e3416d9d610abd0756e6f9 P f8ee67065e464c1b95de6b57be9697ad [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 14:54:27.099386 16261 raft_consensus.cc:482] T 3c2df70ba6e3416d9d610abd0756e6f9 P f8ee67065e464c1b95de6b57be9697ad [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 14:54:27.099457 16261 raft_consensus.cc:3037] T 3c2df70ba6e3416d9d610abd0756e6f9 P f8ee67065e464c1b95de6b57be9697ad [term 0 FOLLOWER]: Advancing to term 1
I20260927 14:54:27.100467 16261 raft_consensus.cc:504] T 3c2df70ba6e3416d9d610abd0756e6f9 P f8ee67065e464c1b95de6b57be9697ad [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9eab78e4338043749a6dbec46092f279" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46285 } } peers { permanent_uuid: "f3e6feeb2c794c849ab9ef7fbef5fb4c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37931 } } peers { permanent_uuid: "f8ee67065e464c1b95de6b57be9697ad" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34745 } }
I20260927 14:54:27.100811 16261 leader_election.cc:288] T 3c2df70ba6e3416d9d610abd0756e6f9 P f8ee67065e464c1b95de6b57be9697ad [CANDIDATE]: Term 1 election: Requested vote from peers 9eab78e4338043749a6dbec46092f279 (127.0.98.3:46285), f3e6feeb2c794c849ab9ef7fbef5fb4c (127.0.98.2:37931)
I20260927 14:54:27.101181 16056 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c2df70ba6e3416d9d610abd0756e6f9" candidate_uuid: "f8ee67065e464c1b95de6b57be9697ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f3e6feeb2c794c849ab9ef7fbef5fb4c"
I20260927 14:54:27.101212 16190 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c2df70ba6e3416d9d610abd0756e6f9" candidate_uuid: "f8ee67065e464c1b95de6b57be9697ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9eab78e4338043749a6dbec46092f279"
I20260927 14:54:27.101308 16056 raft_consensus.cc:3037] T 3c2df70ba6e3416d9d610abd0756e6f9 P f3e6feeb2c794c849ab9ef7fbef5fb4c [term 0 FOLLOWER]: Advancing to term 1
I20260927 14:54:27.101329 16190 raft_consensus.cc:3037] T 3c2df70ba6e3416d9d610abd0756e6f9 P 9eab78e4338043749a6dbec46092f279 [term 0 FOLLOWER]: Advancing to term 1
I20260927 14:54:27.102568 16190 raft_consensus.cc:2454] T 3c2df70ba6e3416d9d610abd0756e6f9 P 9eab78e4338043749a6dbec46092f279 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f8ee67065e464c1b95de6b57be9697ad in term 1.
I20260927 14:54:27.102568 16056 raft_consensus.cc:2454] T 3c2df70ba6e3416d9d610abd0756e6f9 P f3e6feeb2c794c849ab9ef7fbef5fb4c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f8ee67065e464c1b95de6b57be9697ad in term 1.
I20260927 14:54:27.102862 15857 leader_election.cc:302] T 3c2df70ba6e3416d9d610abd0756e6f9 P f8ee67065e464c1b95de6b57be9697ad [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9eab78e4338043749a6dbec46092f279, f8ee67065e464c1b95de6b57be9697ad; no voters:
I20260927 14:54:27.103086 16261 raft_consensus.cc:2781] T 3c2df70ba6e3416d9d610abd0756e6f9 P f8ee67065e464c1b95de6b57be9697ad [term 1 FOLLOWER]: Leader election won for term 1
I20260927 14:54:27.103324 16261 raft_consensus.cc:686] T 3c2df70ba6e3416d9d610abd0756e6f9 P f8ee67065e464c1b95de6b57be9697ad [term 1 LEADER]: Becoming Leader. State: Replica: f8ee67065e464c1b95de6b57be9697ad, State: Running, Role: LEADER
I20260927 14:54:27.103657 16261 consensus_queue.cc:237] T 3c2df70ba6e3416d9d610abd0756e6f9 P f8ee67065e464c1b95de6b57be9697ad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9eab78e4338043749a6dbec46092f279" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46285 } } peers { permanent_uuid: "f3e6feeb2c794c849ab9ef7fbef5fb4c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37931 } } peers { permanent_uuid: "f8ee67065e464c1b95de6b57be9697ad" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34745 } }
I20260927 14:54:27.106753 15773 catalog_manager.cc:5495] T 3c2df70ba6e3416d9d610abd0756e6f9 P f8ee67065e464c1b95de6b57be9697ad reported cstate change: term changed from 0 to 1, leader changed from <none> to f8ee67065e464c1b95de6b57be9697ad (127.0.98.1). New cstate: current_term: 1 leader_uuid: "f8ee67065e464c1b95de6b57be9697ad" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9eab78e4338043749a6dbec46092f279" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46285 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f3e6feeb2c794c849ab9ef7fbef5fb4c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37931 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f8ee67065e464c1b95de6b57be9697ad" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34745 } health_report { overall_health: HEALTHY } } }
I20260927 14:54:27.122638 16252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:54:27.119024 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35886 (local address 127.0.98.1:34745)
0927 14:54:27.119179 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0927 14:54:27.119183 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 14:54:27.119197 (+ 14us) server_negotiation.cc:407] Connection header received
0927 14:54:27.119356 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:54:27.119359 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:54:27.119423 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:54:27.119532 (+ 109us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 14:54:27.119772 (+ 240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:54:27.120279 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:54:27.121205 (+ 926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:54:27.121433 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:54:27.121546 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:54:27.121850 (+ 304us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 14:54:27.122237 (+ 387us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 14:54:27.122367 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0927 14:54:27.122430 (+ 63us) server_negotiation.cc:299] Negotiation successful
0927 14:54:27.122484 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260927 14:54:27.127990 16056 raft_consensus.cc:1261] T 3c2df70ba6e3416d9d610abd0756e6f9 P f3e6feeb2c794c849ab9ef7fbef5fb4c [term 1 FOLLOWER]: Refusing update from remote peer f8ee67065e464c1b95de6b57be9697ad: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 14:54:27.127990 16190 raft_consensus.cc:1261] T 3c2df70ba6e3416d9d610abd0756e6f9 P 9eab78e4338043749a6dbec46092f279 [term 1 FOLLOWER]: Refusing update from remote peer f8ee67065e464c1b95de6b57be9697ad: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 14:54:27.128683 16264 consensus_queue.cc:1035] T 3c2df70ba6e3416d9d610abd0756e6f9 P f8ee67065e464c1b95de6b57be9697ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "f3e6feeb2c794c849ab9ef7fbef5fb4c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37931 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 14:54:27.128870 16261 consensus_queue.cc:1035] T 3c2df70ba6e3416d9d610abd0756e6f9 P f8ee67065e464c1b95de6b57be9697ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "9eab78e4338043749a6dbec46092f279" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 14:54:27.135777 16267 mvcc.cc:205] Tried to move back new op lower bound from 7333973471749754880 to 7333973471659151360. Current Snapshot: MvccSnapshot[applied={T|T < 7333973471749754880}]
I20260927 14:54:27.137102 16268 mvcc.cc:205] Tried to move back new op lower bound from 7333973471749754880 to 7333973471659151360. Current Snapshot: MvccSnapshot[applied={T|T < 7333973471749754880}]
I20260927 14:54:27.144470 15773 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51820:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 14:54:27.144606 15773 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51820:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 14:54:27.146962 15773 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a992827594d14e428ab1ee64535827e5: Sending DeleteTablet for 3 replicas of tablet 3c2df70ba6e3416d9d610abd0756e6f9
I20260927 14:54:27.147698 16170 tablet_service.cc:1514] Processing DeleteTablet for tablet 3c2df70ba6e3416d9d610abd0756e6f9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 14:54:27 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:60798
I20260927 14:54:27.147758 16036 tablet_service.cc:1514] Processing DeleteTablet for tablet 3c2df70ba6e3416d9d610abd0756e6f9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 14:54:27 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:54160
I20260927 14:54:27.147801 15902 tablet_service.cc:1514] Processing DeleteTablet for tablet 3c2df70ba6e3416d9d610abd0756e6f9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 14:54:27 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:35870
I20260927 14:54:27.148175 16276 tablet_replica.cc:330] T 3c2df70ba6e3416d9d610abd0756e6f9 P 9eab78e4338043749a6dbec46092f279: stopping tablet replica
I20260927 14:54:27.148394 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6936m5/build/debug/bin/kudu with pid 15839
I20260927 14:54:27.148479 16276 raft_consensus.cc:2229] T 3c2df70ba6e3416d9d610abd0756e6f9 P 9eab78e4338043749a6dbec46092f279 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 14:54:27.149266 16276 raft_consensus.cc:2258] T 3c2df70ba6e3416d9d610abd0756e6f9 P 9eab78e4338043749a6dbec46092f279 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 14:54:27.150154 16278 tablet_replica.cc:330] T 3c2df70ba6e3416d9d610abd0756e6f9 P f3e6feeb2c794c849ab9ef7fbef5fb4c: stopping tablet replica
I20260927 14:54:27.150445 16278 raft_consensus.cc:2229] T 3c2df70ba6e3416d9d610abd0756e6f9 P f3e6feeb2c794c849ab9ef7fbef5fb4c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 14:54:27.150710 16278 raft_consensus.cc:2258] T 3c2df70ba6e3416d9d610abd0756e6f9 P f3e6feeb2c794c849ab9ef7fbef5fb4c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 14:54:27.150825 16276 ts_tablet_manager.cc:1905] T 3c2df70ba6e3416d9d610abd0756e6f9 P 9eab78e4338043749a6dbec46092f279: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 14:54:27.151975 16278 ts_tablet_manager.cc:1905] T 3c2df70ba6e3416d9d610abd0756e6f9 P f3e6feeb2c794c849ab9ef7fbef5fb4c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 14:54:27.153501 16276 ts_tablet_manager.cc:1918] T 3c2df70ba6e3416d9d610abd0756e6f9 P 9eab78e4338043749a6dbec46092f279: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 14:54:27.153600 16276 log.cc:1196] T 3c2df70ba6e3416d9d610abd0756e6f9 P 9eab78e4338043749a6dbec46092f279: Deleting WAL directory at /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790520684602251-392-0/minicluster-data/ts-2/wal/wals/3c2df70ba6e3416d9d610abd0756e6f9
I20260927 14:54:27.153982 16276 ts_tablet_manager.cc:1939] T 3c2df70ba6e3416d9d610abd0756e6f9 P 9eab78e4338043749a6dbec46092f279: Deleting consensus metadata
I20260927 14:54:27.154655 16278 ts_tablet_manager.cc:1918] T 3c2df70ba6e3416d9d610abd0756e6f9 P f3e6feeb2c794c849ab9ef7fbef5fb4c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 14:54:27.154749 16278 log.cc:1196] T 3c2df70ba6e3416d9d610abd0756e6f9 P f3e6feeb2c794c849ab9ef7fbef5fb4c: Deleting WAL directory at /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790520684602251-392-0/minicluster-data/ts-1/wal/wals/3c2df70ba6e3416d9d610abd0756e6f9
I20260927 14:54:27.155109 16278 ts_tablet_manager.cc:1939] T 3c2df70ba6e3416d9d610abd0756e6f9 P f3e6feeb2c794c849ab9ef7fbef5fb4c: Deleting consensus metadata
I20260927 14:54:27.155267 15763 catalog_manager.cc:4841] TS 9eab78e4338043749a6dbec46092f279 (127.0.98.3:46285): tablet 3c2df70ba6e3416d9d610abd0756e6f9 (table test-table [id=3866d04318874c01ae4d89b5f00c30a2]) successfully deleted
I20260927 14:54:27.156020 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6936m5/build/debug/bin/kudu with pid 15973
W20260927 14:54:27.156091 15762 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:34745 (error 108)
I20260927 14:54:27.156509 15764 catalog_manager.cc:4841] TS f3e6feeb2c794c849ab9ef7fbef5fb4c (127.0.98.2:37931): tablet 3c2df70ba6e3416d9d610abd0756e6f9 (table test-table [id=3866d04318874c01ae4d89b5f00c30a2]) successfully deleted
I20260927 14:54:27.157413 16250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:54:27.157018 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:34745 (local address 127.0.0.1:35896)
0927 14:54:27.157244 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0927 14:54:27.157317 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:34745: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":130}
W20260927 14:54:27.157584 15762 catalog_manager.cc:4584] TS f8ee67065e464c1b95de6b57be9697ad (127.0.98.1:34745): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 3c2df70ba6e3416d9d610abd0756e6f9: Network error: Client connection negotiation failed: client connection to 127.0.98.1:34745: connect: Connection refused (error 111)
I20260927 14:54:27.161224 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6936m5/build/debug/bin/kudu with pid 16107
I20260927 14:54:27.166898 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6936m5/build/debug/bin/kudu with pid 15745
I20260927 14:54:47.173382 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6936m5/build/debug/bin/kudu
/tmp/dist-test-task6936m5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790520684602251-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790520684602251-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790520684602251-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790520684602251-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34779
--webserver_interface=127.0.98.62
--webserver_port=44315
--keytab_file=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790520684602251-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40871
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:34779
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790520684602251-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790520684602251-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790520684602251-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 14:54:47.283111 16280 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 14:54:47.283385 16280 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 14:54:47.283469 16280 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 14:54:47.287546 16280 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260927 14:54:47.287618 16280 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260927 14:54:47.287638 16280 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260927 14:54:47.287655 16280 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 14:54:47.287714 16280 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 14:54:47.287755 16280 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 14:54:47.287788 16280 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 14:54:47.287828 16280 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 14:54:47.293244 16280 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40871
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790520684602251-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790520684602251-392-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:34779
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790520684602251-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34779
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790520684602251-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=44315
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16280
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790520684602251-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:15:17 UTC on ccycloud.quasar-ledxfe.root.comops.site
build id 4676134
I20260927 14:54:47.294303 16280 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 14:54:47.295336 16280 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 14:54:47.309222 16280 server_base.cc:1034] running on GCE node
W20260927 14:54:47.309161 16289 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 14:54:47.309163 16287 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 14:54:47.309161 16286 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 14:54:47.310235 16280 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 14:54:47.311390 16280 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 14:54:47.312562 16280 hybrid_clock.cc:648] HybridClock initialized: now 1790520887312544 us; error 34 us; skew 500 ppm
Sep 27 14:54:47 dist-test-slave-xk1g krb5kdc[15732](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790520887, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 14:54:47.319921 16280 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 14:54:47.320891 16280 webserver.cc:460] Webserver started at http://127.0.98.62:44315/ using document root <none> and password file <none>
I20260927 14:54:47.321570 16280 fs_manager.cc:359] Metadata directory not provided
I20260927 14:54:47.321686 16280 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 14:54:47.325621 16280 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.004s sys 0.000s
I20260927 14:54:47.327735 16296 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 14:54:47.328941 16280 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 14:54:47.329123 16280 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790520684602251-392-0/minicluster-data/master-0/data,/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790520684602251-392-0/minicluster-data/master-0/wal
uuid: "a992827594d14e428ab1ee64535827e5"
format_stamp: "Formatted at 2026-09-27 14:54:26 on dist-test-slave-xk1g"
server_key: "d50aee429206bc58e096762fd7248ed8"
server_key_iv: "82418a00305e96599705060c600e3e83"
server_key_version: "encryptionkey@0"
I20260927 14:54:47.329700 16280 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790520684602251-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790520684602251-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790520684602251-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 14:54:47.351104 16280 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 14:54:47.355247 16280 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 14:54:47.355448 16280 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 14:54:47.362594 16280 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:34779
I20260927 14:54:47.362824 16348 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:34779 every 8 connection(s)
I20260927 14:54:47.363723 16280 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790520684602251-392-0/minicluster-data/master-0/data/info.pb
I20260927 14:54:47.368005 16349 sys_catalog.cc:263] Verifying existing consensus state
I20260927 14:54:47.369588 16349 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a992827594d14e428ab1ee64535827e5: Bootstrap starting.
I20260927 14:54:47.369858 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6936m5/build/debug/bin/kudu as pid 16280
I20260927 14:54:47.370271 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6936m5/build/debug/bin/kudu
/tmp/dist-test-task6936m5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790520684602251-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790520684602251-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790520684602251-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790520684602251-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:34745
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:34779
--webserver_port=40025
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790520684602251-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40871
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790520684602251-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790520684602251-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790520684602251-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 14:54:47.387094 16349 log.cc:824] T 00000000000000000000000000000000 P a992827594d14e428ab1ee64535827e5: Log is configured to *not* fsync() on all Append() calls
I20260927 14:54:47.395571 16349 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a992827594d14e428ab1ee64535827e5: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 14:54:47.395949 16349 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a992827594d14e428ab1ee64535827e5: Bootstrap complete.
I20260927 14:54:47.399959 16349 raft_consensus.cc:348] T 00000000000000000000000000000000 P a992827594d14e428ab1ee64535827e5 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a992827594d14e428ab1ee64535827e5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34779 } }
I20260927 14:54:47.400339 16349 raft_consensus.cc:729] T 00000000000000000000000000000000 P a992827594d14e428ab1ee64535827e5 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: a992827594d14e428ab1ee64535827e5, State: Initialized, Role: FOLLOWER
I20260927 14:54:47.400872 16349 consensus_queue.cc:260] T 00000000000000000000000000000000 P a992827594d14e428ab1ee64535827e5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a992827594d14e428ab1ee64535827e5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34779 } }
I20260927 14:54:47.400992 16349 raft_consensus.cc:388] T 00000000000000000000000000000000 P a992827594d14e428ab1ee64535827e5 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 14:54:47.401103 16349 raft_consensus.cc:482] T 00000000000000000000000000000000 P a992827594d14e428ab1ee64535827e5 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 14:54:47.401221 16349 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a992827594d14e428ab1ee64535827e5 [term 1 FOLLOWER]: Advancing to term 2
I20260927 14:54:47.403054 16349 raft_consensus.cc:504] T 00000000000000000000000000000000 P a992827594d14e428ab1ee64535827e5 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a992827594d14e428ab1ee64535827e5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34779 } }
I20260927 14:54:47.403394 16349 leader_election.cc:302] T 00000000000000000000000000000000 P a992827594d14e428ab1ee64535827e5 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a992827594d14e428ab1ee64535827e5; no voters:
I20260927 14:54:47.403728 16349 leader_election.cc:288] T 00000000000000000000000000000000 P a992827594d14e428ab1ee64535827e5 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260927 14:54:47.403818 16353 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a992827594d14e428ab1ee64535827e5 [term 2 FOLLOWER]: Leader election won for term 2
I20260927 14:54:47.404086 16353 raft_consensus.cc:686] T 00000000000000000000000000000000 P a992827594d14e428ab1ee64535827e5 [term 2 LEADER]: Becoming Leader. State: Replica: a992827594d14e428ab1ee64535827e5, State: Running, Role: LEADER
I20260927 14:54:47.404425 16353 consensus_queue.cc:237] T 00000000000000000000000000000000 P a992827594d14e428ab1ee64535827e5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a992827594d14e428ab1ee64535827e5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34779 } }
I20260927 14:54:47.404956 16349 sys_catalog.cc:564] T 00000000000000000000000000000000 P a992827594d14e428ab1ee64535827e5 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 14:54:47.405357 16353 sys_catalog.cc:455] T 00000000000000000000000000000000 P a992827594d14e428ab1ee64535827e5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a992827594d14e428ab1ee64535827e5. Latest consensus state: current_term: 2 leader_uuid: "a992827594d14e428ab1ee64535827e5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a992827594d14e428ab1ee64535827e5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34779 } } }
I20260927 14:54:47.405558 16353 sys_catalog.cc:458] T 00000000000000000000000000000000 P a992827594d14e428ab1ee64535827e5 [sys.catalog]: This master's current role is: LEADER
I20260927 14:54:47.405721 16354 sys_catalog.cc:455] T 00000000000000000000000000000000 P a992827594d14e428ab1ee64535827e5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "a992827594d14e428ab1ee64535827e5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a992827594d14e428ab1ee64535827e5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34779 } } }
I20260927 14:54:47.405846 16354 sys_catalog.cc:458] T 00000000000000000000000000000000 P a992827594d14e428ab1ee64535827e5 [sys.catalog]: This master's current role is: LEADER
I20260927 14:54:47.406113 16357 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 14:54:47.411302 16357 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 14:54:47.412134 16357 catalog_manager.cc:1259] Loaded cluster ID: d488d1bfd5274fd3ab35889a54cbc5d0
I20260927 14:54:47.412205 16357 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 14:54:47.414764 16357 catalog_manager.cc:1504] Loading token signing keys...
I20260927 14:54:47.415509 16357 catalog_manager.cc:5879] T 00000000000000000000000000000000 P a992827594d14e428ab1ee64535827e5: Loaded TSK: 0
I20260927 14:54:47.435200 16357 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a992827594d14e428ab1ee64535827e5: Generated new TSK 1
I20260927 14:54:47.435480 16357 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260927 14:54:47.483103 16351 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 14:54:47.483366 16351 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 14:54:47.483448 16351 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 14:54:47.487445 16351 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 14:54:47.487515 16351 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 14:54:47.487593 16351 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 14:54:47.492632 16351 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40871
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790520684602251-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790520684602251-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790520684602251-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:34745
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790520684602251-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=40025
--tserver_master_addrs=127.0.98.62:34779
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16351
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790520684602251-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:15:17 UTC on ccycloud.quasar-ledxfe.root.comops.site
build id 4676134
I20260927 14:54:47.493633 16351 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 14:54:47.494576 16351 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 14:54:47.503544 16377 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 14:54:47.503580 16379 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 14:54:47.503572 16376 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 14:54:47.504268 16351 server_base.cc:1034] running on GCE node
I20260927 14:54:47.504681 16351 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 14:54:47.505395 16351 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 14:54:47.506568 16351 hybrid_clock.cc:648] HybridClock initialized: now 1790520887506540 us; error 46 us; skew 500 ppm
Sep 27 14:54:47 dist-test-slave-xk1g krb5kdc[15732](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790520887, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 14:54:47.509517 16351 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 14:54:47.510555 16351 webserver.cc:460] Webserver started at http://127.0.98.1:40025/ using document root <none> and password file <none>
I20260927 14:54:47.511242 16351 fs_manager.cc:359] Metadata directory not provided
I20260927 14:54:47.511363 16351 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 14:54:47.516215 16351 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260927 14:54:47.518834 16386 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 14:54:47.520253 16351 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260927 14:54:47.520370 16351 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790520684602251-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790520684602251-392-0/minicluster-data/ts-0/wal
uuid: "f8ee67065e464c1b95de6b57be9697ad"
format_stamp: "Formatted at 2026-09-27 14:54:26 on dist-test-slave-xk1g"
server_key: "952ee1f62b1ebc8e55e8c976234b4dce"
server_key_iv: "592e8d010d72b7b41a99653c5bc2d2e6"
server_key_version: "encryptionkey@0"
I20260927 14:54:47.521090 16351 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790520684602251-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790520684602251-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790520684602251-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 14:54:47.569720 16351 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 14:54:47.573035 16351 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 14:54:47.573210 16351 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 14:54:47.573747 16351 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 14:54:47.574750 16393 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260927 14:54:47.578835 16351 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260927 14:54:47.578894 16351 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.004s user 0.000s sys 0.000s
I20260927 14:54:47.578971 16351 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260927 14:54:47.580560 16351 ts_tablet_manager.cc:610] Registered 1 tablets
I20260927 14:54:47.580606 16351 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260927 14:54:47.580876 16393 tablet_bootstrap.cc:492] T 3c2df70ba6e3416d9d610abd0756e6f9 P f8ee67065e464c1b95de6b57be9697ad: Bootstrap starting.
I20260927 14:54:47.591579 16351 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:34745
I20260927 14:54:47.591675 16500 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:34745 every 8 connection(s)
I20260927 14:54:47.592608 16351 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790520684602251-392-0/minicluster-data/ts-0/data/info.pb
I20260927 14:54:47.597007 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6936m5/build/debug/bin/kudu as pid 16351
I20260927 14:54:47.597527 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6936m5/build/debug/bin/kudu
/tmp/dist-test-task6936m5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790520684602251-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790520684602251-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790520684602251-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790520684602251-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:37931
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:34779
--webserver_port=33621
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790520684602251-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40871
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790520684602251-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790520684602251-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790520684602251-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 14:54:47.599639 16393 log.cc:824] T 3c2df70ba6e3416d9d610abd0756e6f9 P f8ee67065e464c1b95de6b57be9697ad: Log is configured to *not* fsync() on all Append() calls
Sep 27 14:54:47 dist-test-slave-xk1g krb5kdc[15732](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790520887, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 14:54:47.608176 16393 tablet_bootstrap.cc:492] T 3c2df70ba6e3416d9d610abd0756e6f9 P f8ee67065e464c1b95de6b57be9697ad: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 14:54:47.608625 16393 tablet_bootstrap.cc:492] T 3c2df70ba6e3416d9d610abd0756e6f9 P f8ee67065e464c1b95de6b57be9697ad: Bootstrap complete.
I20260927 14:54:47.609944 16393 ts_tablet_manager.cc:1397] T 3c2df70ba6e3416d9d610abd0756e6f9 P f8ee67065e464c1b95de6b57be9697ad: Time spent bootstrapping tablet: real 0.029s user 0.014s sys 0.014s
I20260927 14:54:47.612392 16503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:54:47.594714 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:60753 (local address 127.0.98.62:34779)
0927 14:54:47.595098 (+ 384us) server_negotiation.cc:206] Beginning negotiation
0927 14:54:47.595118 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0927 14:54:47.595492 (+ 374us) server_negotiation.cc:407] Connection header received
0927 14:54:47.597047 (+ 1555us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:54:47.597090 (+ 43us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:54:47.597532 (+ 442us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:54:47.597902 (+ 370us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:54:47.599324 (+ 1422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:54:47.600664 (+ 1340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:54:47.602315 (+ 1651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:54:47.603142 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:54:47.603353 (+ 211us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:54:47.606518 (+ 3165us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:54:47.606525 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:54:47.606541 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 14:54:47.606574 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:54:47.608755 (+ 2181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:54:47.609469 (+ 714us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:54:47.609478 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:54:47.609486 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:54:47.609636 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:54:47.610041 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:54:47.610045 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:54:47.610046 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:54:47.610497 (+ 451us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:54:47.610633 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0927 14:54:47.611257 (+ 624us) server_negotiation.cc:299] Negotiation successful
0927 14:54:47.611556 (+ 299us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":238,"thread_start_us":115,"threads_started":1}
I20260927 14:54:47.612409 16504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:54:47.594694 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34779 (local address 127.0.98.1:60753)
0927 14:54:47.595284 (+ 590us) negotiation.cc:107] Waiting for socket to connect
0927 14:54:47.595353 (+ 69us) client_negotiation.cc:174] Beginning negotiation
0927 14:54:47.596542 (+ 1189us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:54:47.598020 (+ 1478us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:54:47.598029 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:54:47.598492 (+ 463us) client_negotiation.cc:189] Negotiated authn=SASL
0927 14:54:47.599101 (+ 609us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:54:47.599121 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:54:47.600821 (+ 1700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:54:47.600827 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:54:47.602167 (+ 1340us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:54:47.602178 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:54:47.603312 (+ 1134us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:54:47.603316 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:54:47.603492 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:54:47.604208 (+ 716us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 14:54:47.604227 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0927 14:54:47.606275 (+ 2048us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 14:54:47.608897 (+ 2622us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 14:54:47.608906 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 14:54:47.608922 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0927 14:54:47.609350 (+ 428us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 14:54:47.609751 (+ 401us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 14:54:47.609754 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 14:54:47.609756 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 14:54:47.609920 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 14:54:47.610619 (+ 699us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 14:54:47.610625 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 14:54:47.610908 (+ 283us) client_negotiation.cc:769] Sending connection context
0927 14:54:47.611086 (+ 178us) client_negotiation.cc:240] Negotiation successful
0927 14:54:47.611410 (+ 324us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":365,"thread_start_us":142,"threads_started":1}
I20260927 14:54:47.612816 16393 raft_consensus.cc:348] T 3c2df70ba6e3416d9d610abd0756e6f9 P f8ee67065e464c1b95de6b57be9697ad [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9eab78e4338043749a6dbec46092f279" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46285 } } peers { permanent_uuid: "f3e6feeb2c794c849ab9ef7fbef5fb4c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37931 } } peers { permanent_uuid: "f8ee67065e464c1b95de6b57be9697ad" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34745 } }
I20260927 14:54:47.613338 16393 raft_consensus.cc:729] T 3c2df70ba6e3416d9d610abd0756e6f9 P f8ee67065e464c1b95de6b57be9697ad [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: f8ee67065e464c1b95de6b57be9697ad, State: Initialized, Role: FOLLOWER
I20260927 14:54:47.613843 16393 consensus_queue.cc:260] T 3c2df70ba6e3416d9d610abd0756e6f9 P f8ee67065e464c1b95de6b57be9697ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9eab78e4338043749a6dbec46092f279" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46285 } } peers { permanent_uuid: "f3e6feeb2c794c849ab9ef7fbef5fb4c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37931 } } peers { permanent_uuid: "f8ee67065e464c1b95de6b57be9697ad" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34745 } }
I20260927 14:54:47.614861 16393 ts_tablet_manager.cc:1428] T 3c2df70ba6e3416d9d610abd0756e6f9 P f8ee67065e464c1b95de6b57be9697ad: Time spent starting tablet: real 0.005s user 0.004s sys 0.001s
I20260927 14:54:47.615154 16501 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34779
I20260927 14:54:47.615391 16501 heartbeater.cc:464] Registering TS with master...
I20260927 14:54:47.615854 16501 heartbeater.cc:511] Master 127.0.98.62:34779 requested a full tablet report, sending...
I20260927 14:54:47.618182 16313 ts_manager.cc:194] Registered new tserver with Master: f8ee67065e464c1b95de6b57be9697ad (127.0.98.1:34745)
I20260927 14:54:47.621417 16313 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:60753
I20260927 14:54:47.623754 16501 heartbeater.cc:503] Master 127.0.98.62:34779 was elected leader, sending a full tablet report...
Sep 27 14:54:47 dist-test-slave-xk1g krb5kdc[15732](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790520887, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260927 14:54:47.631320 16509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:54:47.620944 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43598 (local address 127.0.98.1:34745)
0927 14:54:47.621613 (+ 669us) server_negotiation.cc:206] Beginning negotiation
0927 14:54:47.621623 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 14:54:47.621646 (+ 23us) server_negotiation.cc:407] Connection header received
0927 14:54:47.621713 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:54:47.621718 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:54:47.621906 (+ 188us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:54:47.622117 (+ 211us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:54:47.622721 (+ 604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:54:47.623771 (+ 1050us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:54:47.624957 (+ 1186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:54:47.625438 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:54:47.625523 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:54:47.627846 (+ 2323us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:54:47.627855 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:54:47.627867 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 14:54:47.627896 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:54:47.629456 (+ 1560us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:54:47.629924 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:54:47.629930 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:54:47.629932 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:54:47.630015 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:54:47.630307 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:54:47.630312 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:54:47.630314 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:54:47.630543 (+ 229us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:54:47.630684 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0927 14:54:47.631021 (+ 337us) server_negotiation.cc:299] Negotiation successful
0927 14:54:47.631181 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":579,"thread_start_us":121,"threads_started":1}
I20260927 14:54:47.631317 16508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:54:47.620833 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:34745 (local address 127.0.0.1:43598)
0927 14:54:47.621423 (+ 590us) negotiation.cc:107] Waiting for socket to connect
0927 14:54:47.621466 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0927 14:54:47.621591 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:54:47.622067 (+ 476us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:54:47.622073 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:54:47.622448 (+ 375us) client_negotiation.cc:189] Negotiated authn=SASL
0927 14:54:47.622595 (+ 147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:54:47.622604 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:54:47.623918 (+ 1314us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:54:47.623921 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:54:47.624828 (+ 907us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:54:47.624839 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:54:47.625535 (+ 696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:54:47.625551 (+ 16us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:54:47.625678 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:54:47.626186 (+ 508us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 14:54:47.626200 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0927 14:54:47.627704 (+ 1504us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 14:54:47.629564 (+ 1860us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 14:54:47.629575 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 14:54:47.629580 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 14:54:47.629811 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 14:54:47.630116 (+ 305us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 14:54:47.630119 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 14:54:47.630122 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 14:54:47.630208 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 14:54:47.630659 (+ 451us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 14:54:47.630665 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 14:54:47.630791 (+ 126us) client_negotiation.cc:769] Sending connection context
0927 14:54:47.630929 (+ 138us) client_negotiation.cc:240] Negotiation successful
0927 14:54:47.631124 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":421,"thread_start_us":112,"threads_started":1}
I20260927 14:54:47.633117 16435 tablet_service.cc:1514] Processing DeleteTablet for tablet 3c2df70ba6e3416d9d610abd0756e6f9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 14:54:27 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:43598
I20260927 14:54:47.634073 16510 tablet_replica.cc:330] stopping tablet replica
I20260927 14:54:47.634402 16510 raft_consensus.cc:2229] T 3c2df70ba6e3416d9d610abd0756e6f9 P f8ee67065e464c1b95de6b57be9697ad [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 14:54:47.634688 16510 raft_consensus.cc:2258] T 3c2df70ba6e3416d9d610abd0756e6f9 P f8ee67065e464c1b95de6b57be9697ad [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 14:54:47.635766 16510 ts_tablet_manager.cc:1905] T 3c2df70ba6e3416d9d610abd0756e6f9 P f8ee67065e464c1b95de6b57be9697ad: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 14:54:47.640118 16510 ts_tablet_manager.cc:1918] T 3c2df70ba6e3416d9d610abd0756e6f9 P f8ee67065e464c1b95de6b57be9697ad: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 14:54:47.640224 16510 log.cc:1196] T 3c2df70ba6e3416d9d610abd0756e6f9 P f8ee67065e464c1b95de6b57be9697ad: Deleting WAL directory at /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790520684602251-392-0/minicluster-data/ts-0/wal/wals/3c2df70ba6e3416d9d610abd0756e6f9
I20260927 14:54:47.640522 16510 ts_tablet_manager.cc:1939] T 3c2df70ba6e3416d9d610abd0756e6f9 P f8ee67065e464c1b95de6b57be9697ad: Deleting consensus metadata
I20260927 14:54:47.642624 16298 catalog_manager.cc:4841] TS f8ee67065e464c1b95de6b57be9697ad (127.0.98.1:34745): tablet 3c2df70ba6e3416d9d610abd0756e6f9 (table test-table [id=3866d04318874c01ae4d89b5f00c30a2]) successfully deleted
W20260927 14:54:47.717137 16505 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 14:54:47.717607 16505 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 14:54:47.717685 16505 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 14:54:47.721841 16505 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 14:54:47.721900 16505 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 14:54:47.721959 16505 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 14:54:47.727101 16505 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40871
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790520684602251-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790520684602251-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790520684602251-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:37931
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790520684602251-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=33621
--tserver_master_addrs=127.0.98.62:34779
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16505
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790520684602251-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:15:17 UTC on ccycloud.quasar-ledxfe.root.comops.site
build id 4676134
I20260927 14:54:47.728094 16505 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 14:54:47.729040 16505 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 14:54:47.738364 16516 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 14:54:47.738484 16505 server_base.cc:1034] running on GCE node
W20260927 14:54:47.738332 16518 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 14:54:47.738324 16515 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 14:54:47.739110 16505 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 14:54:47.739816 16505 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 14:54:47.740971 16505 hybrid_clock.cc:648] HybridClock initialized: now 1790520887740942 us; error 44 us; skew 500 ppm
Sep 27 14:54:47 dist-test-slave-xk1g krb5kdc[15732](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790520887, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 14:54:47.743590 16505 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 14:54:47.744460 16505 webserver.cc:460] Webserver started at http://127.0.98.2:33621/ using document root <none> and password file <none>
I20260927 14:54:47.744971 16505 fs_manager.cc:359] Metadata directory not provided
I20260927 14:54:47.745038 16505 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 14:54:47.748649 16505 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.000s sys 0.002s
I20260927 14:54:47.750608 16525 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 14:54:47.751766 16505 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 14:54:47.751910 16505 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790520684602251-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790520684602251-392-0/minicluster-data/ts-1/wal
uuid: "f3e6feeb2c794c849ab9ef7fbef5fb4c"
format_stamp: "Formatted at 2026-09-27 14:54:26 on dist-test-slave-xk1g"
server_key: "d4aafd3368d18eac59a1a1b0b80dca96"
server_key_iv: "bd37185f913e046786f09f6fba5e5470"
server_key_version: "encryptionkey@0"
I20260927 14:54:47.752444 16505 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790520684602251-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790520684602251-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790520684602251-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 14:54:47.788338 16505 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 14:54:47.791445 16505 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 14:54:47.791664 16505 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 14:54:47.792286 16505 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 14:54:47.793183 16505 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 14:54:47.793255 16505 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 14:54:47.793324 16505 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 14:54:47.793368 16505 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 14:54:47.802423 16505 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:37931
I20260927 14:54:47.802448 16638 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:37931 every 8 connection(s)
I20260927 14:54:47.803438 16505 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790520684602251-392-0/minicluster-data/ts-1/data/info.pb
I20260927 14:54:47.804958 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6936m5/build/debug/bin/kudu as pid 16505
I20260927 14:54:47.805562 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6936m5/build/debug/bin/kudu
/tmp/dist-test-task6936m5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790520684602251-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790520684602251-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790520684602251-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790520684602251-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:46285
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:34779
--webserver_port=45735
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790520684602251-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40871
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790520684602251-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790520684602251-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790520684602251-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 14:54:47 dist-test-slave-xk1g krb5kdc[15732](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790520887, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 14:54:47.818692 16503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:54:47.805412 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:40217 (local address 127.0.98.62:34779)
0927 14:54:47.805574 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0927 14:54:47.805578 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 14:54:47.806656 (+ 1078us) server_negotiation.cc:407] Connection header received
0927 14:54:47.808037 (+ 1381us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:54:47.808041 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:54:47.808096 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:54:47.808206 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:54:47.809510 (+ 1304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:54:47.810050 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:54:47.811346 (+ 1296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:54:47.811588 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:54:47.811702 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:54:47.814640 (+ 2938us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:54:47.814643 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:54:47.814646 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 14:54:47.814677 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:54:47.816339 (+ 1662us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:54:47.816902 (+ 563us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:54:47.816906 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:54:47.816908 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:54:47.816994 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:54:47.817458 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:54:47.817461 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:54:47.817463 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:54:47.817664 (+ 201us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:54:47.817775 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0927 14:54:47.818372 (+ 597us) server_negotiation.cc:299] Negotiation successful
0927 14:54:47.818509 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260927 14:54:47.819599 16641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:54:47.805964 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34779 (local address 127.0.98.2:40217)
0927 14:54:47.806481 (+ 517us) negotiation.cc:107] Waiting for socket to connect
0927 14:54:47.806535 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0927 14:54:47.807814 (+ 1279us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:54:47.808402 (+ 588us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:54:47.808409 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:54:47.808777 (+ 368us) client_negotiation.cc:189] Negotiated authn=SASL
0927 14:54:47.809358 (+ 581us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:54:47.809372 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:54:47.810185 (+ 813us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:54:47.810189 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:54:47.811202 (+ 1013us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:54:47.811213 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:54:47.811749 (+ 536us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:54:47.811753 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:54:47.811931 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:54:47.812649 (+ 718us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 14:54:47.812668 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0927 14:54:47.814360 (+ 1692us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 14:54:47.816474 (+ 2114us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 14:54:47.816482 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 14:54:47.816501 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0927 14:54:47.816790 (+ 289us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 14:54:47.817129 (+ 339us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 14:54:47.817134 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 14:54:47.817139 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 14:54:47.817348 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 14:54:47.817775 (+ 427us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 14:54:47.817783 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 14:54:47.818062 (+ 279us) client_negotiation.cc:769] Sending connection context
0927 14:54:47.818272 (+ 210us) client_negotiation.cc:240] Negotiation successful
0927 14:54:47.818633 (+ 361us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":310,"thread_start_us":147,"threads_started":1}
I20260927 14:54:47.820902 16639 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34779
I20260927 14:54:47.821214 16639 heartbeater.cc:464] Registering TS with master...
I20260927 14:54:47.821744 16639 heartbeater.cc:511] Master 127.0.98.62:34779 requested a full tablet report, sending...
I20260927 14:54:47.822839 16313 ts_manager.cc:194] Registered new tserver with Master: f3e6feeb2c794c849ab9ef7fbef5fb4c (127.0.98.2:37931)
I20260927 14:54:47.823596 16313 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:40217
W20260927 14:54:47.922251 16642 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 14:54:47.922500 16642 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 14:54:47.922541 16642 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 14:54:47.926555 16642 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 14:54:47.926612 16642 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 14:54:47.926667 16642 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 14:54:47.931911 16642 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40871
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790520684602251-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790520684602251-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790520684602251-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:46285
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790520684602251-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=45735
--tserver_master_addrs=127.0.98.62:34779
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16642
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790520684602251-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:15:17 UTC on ccycloud.quasar-ledxfe.root.comops.site
build id 4676134
I20260927 14:54:47.932988 16642 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 14:54:47.934124 16642 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 14:54:47.944244 16642 server_base.cc:1034] running on GCE node
W20260927 14:54:47.944198 16650 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 14:54:47.944170 16648 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 14:54:47.944164 16647 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 14:54:47.944954 16642 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 14:54:47.945636 16642 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 14:54:47.946868 16642 hybrid_clock.cc:648] HybridClock initialized: now 1790520887946830 us; error 53 us; skew 500 ppm
Sep 27 14:54:47 dist-test-slave-xk1g krb5kdc[15732](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790520887, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 14:54:47.949674 16642 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 14:54:47.950484 16642 webserver.cc:460] Webserver started at http://127.0.98.3:45735/ using document root <none> and password file <none>
I20260927 14:54:47.951001 16642 fs_manager.cc:359] Metadata directory not provided
I20260927 14:54:47.951071 16642 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 14:54:47.954828 16642 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260927 14:54:47.956844 16657 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 14:54:47.958066 16642 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 14:54:47.958169 16642 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790520684602251-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790520684602251-392-0/minicluster-data/ts-2/wal
uuid: "9eab78e4338043749a6dbec46092f279"
format_stamp: "Formatted at 2026-09-27 14:54:26 on dist-test-slave-xk1g"
server_key: "678491584feb82d497710ade28548c61"
server_key_iv: "acf9c8bba7ec8bb4f65321f6ec1f109c"
server_key_version: "encryptionkey@0"
I20260927 14:54:47.958665 16642 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790520684602251-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790520684602251-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790520684602251-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 14:54:47.984354 16642 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 14:54:47.987671 16642 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 14:54:47.987843 16642 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 14:54:47.988373 16642 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 14:54:47.989164 16642 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 14:54:47.989209 16642 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 14:54:47.989249 16642 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 14:54:47.989264 16642 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 14:54:47.998198 16642 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:46285
I20260927 14:54:47.998219 16770 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:46285 every 8 connection(s)
I20260927 14:54:47.999215 16642 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790520684602251-392-0/minicluster-data/ts-2/data/info.pb
I20260927 14:54:48.008642 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6936m5/build/debug/bin/kudu as pid 16642
Sep 27 14:54:48 dist-test-slave-xk1g krb5kdc[15732](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790520887, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 14:54:48.013531 16503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:54:48.000991 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:35523 (local address 127.0.98.62:34779)
0927 14:54:48.001165 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0927 14:54:48.001169 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 14:54:48.001991 (+ 822us) server_negotiation.cc:407] Connection header received
0927 14:54:48.003315 (+ 1324us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:54:48.003319 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:54:48.003378 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:54:48.003455 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:54:48.004687 (+ 1232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:54:48.005304 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:54:48.006584 (+ 1280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:54:48.006816 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:54:48.006929 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:54:48.009548 (+ 2619us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:54:48.009553 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:54:48.009556 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 14:54:48.009586 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:54:48.011138 (+ 1552us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:54:48.011734 (+ 596us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:54:48.011738 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:54:48.011739 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:54:48.011835 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:54:48.012199 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:54:48.012203 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:54:48.012204 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:54:48.012447 (+ 243us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:54:48.012548 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0927 14:54:48.013182 (+ 634us) server_negotiation.cc:299] Negotiation successful
0927 14:54:48.013332 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49,"spinlock_wait_cycles":2432}
I20260927 14:54:48.014590 16773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:54:48.001407 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34779 (local address 127.0.98.3:35523)
0927 14:54:48.001836 (+ 429us) negotiation.cc:107] Waiting for socket to connect
0927 14:54:48.001878 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0927 14:54:48.003084 (+ 1206us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:54:48.003627 (+ 543us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:54:48.003635 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:54:48.004015 (+ 380us) client_negotiation.cc:189] Negotiated authn=SASL
0927 14:54:48.004520 (+ 505us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:54:48.004542 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:54:48.005428 (+ 886us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:54:48.005431 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:54:48.006463 (+ 1032us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:54:48.006472 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:54:48.006967 (+ 495us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:54:48.006971 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:54:48.007140 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:54:48.007749 (+ 609us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 14:54:48.007765 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 14:54:48.009348 (+ 1583us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 14:54:48.011273 (+ 1925us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 14:54:48.011280 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 14:54:48.011297 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0927 14:54:48.011645 (+ 348us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 14:54:48.011926 (+ 281us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 14:54:48.011928 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 14:54:48.011929 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 14:54:48.012105 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 14:54:48.012543 (+ 438us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 14:54:48.012548 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 14:54:48.012874 (+ 326us) client_negotiation.cc:769] Sending connection context
0927 14:54:48.013113 (+ 239us) client_negotiation.cc:240] Negotiation successful
0927 14:54:48.013452 (+ 339us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":243,"thread_start_us":114,"threads_started":1}
I20260927 14:54:48.015892 16771 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34779
I20260927 14:54:48.016144 16771 heartbeater.cc:464] Registering TS with master...
I20260927 14:54:48.016801 16771 heartbeater.cc:511] Master 127.0.98.62:34779 requested a full tablet report, sending...
I20260927 14:54:48.017930 16313 ts_manager.cc:194] Registered new tserver with Master: 9eab78e4338043749a6dbec46092f279 (127.0.98.3:46285)
I20260927 14:54:48.018095 16775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:54:48.009431 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57678 (local address 127.0.98.62:34779)
0927 14:54:48.009734 (+ 303us) server_negotiation.cc:206] Beginning negotiation
0927 14:54:48.009739 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 14:54:48.009843 (+ 104us) server_negotiation.cc:407] Connection header received
0927 14:54:48.010005 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:54:48.010008 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:54:48.010053 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:54:48.010140 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:54:48.010821 (+ 681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:54:48.011444 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:54:48.012678 (+ 1234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:54:48.012990 (+ 312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:54:48.013111 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:54:48.014651 (+ 1540us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:54:48.014654 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:54:48.014656 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 14:54:48.014700 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:54:48.016263 (+ 1563us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:54:48.016719 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:54:48.016722 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:54:48.016724 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:54:48.016813 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:54:48.017127 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:54:48.017130 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:54:48.017132 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:54:48.017317 (+ 185us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:54:48.017538 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0927 14:54:48.017733 (+ 195us) server_negotiation.cc:299] Negotiation successful
0927 14:54:48.017867 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":232,"thread_start_us":113,"threads_started":1}
I20260927 14:54:48.018724 16313 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:35523
I20260927 14:54:48.019865 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 14:54:48.023695 16775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:54:48.020504 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57680 (local address 127.0.98.62:34779)
0927 14:54:48.020630 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0927 14:54:48.020633 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 14:54:48.020831 (+ 198us) server_negotiation.cc:407] Connection header received
0927 14:54:48.020977 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:54:48.020979 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:54:48.021114 (+ 135us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:54:48.021224 (+ 110us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 14:54:48.021476 (+ 252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:54:48.021964 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:54:48.022737 (+ 773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:54:48.022951 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:54:48.023028 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:54:48.023258 (+ 230us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 14:54:48.023376 (+ 118us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0927 14:54:48.023558 (+ 182us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:57680: token expired
Metrics: {"server-negotiator.queue_time_us":52}
W20260927 14:54:48.023763 16775 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:57680: token expired
W20260927 14:54:48.024088 16239 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.0.98.62:34779)
I20260927 14:54:48.030825 16775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:54:48.024448 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57682 (local address 127.0.98.62:34779)
0927 14:54:48.024565 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0927 14:54:48.024568 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 14:54:48.024579 (+ 11us) server_negotiation.cc:407] Connection header received
0927 14:54:48.024658 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:54:48.024660 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:54:48.024703 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:54:48.024763 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:54:48.025416 (+ 653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:54:48.025929 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:54:48.026666 (+ 737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:54:48.026909 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:54:48.026983 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:54:48.028224 (+ 1241us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:54:48.028226 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:54:48.028228 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 14:54:48.028248 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:54:48.029522 (+ 1274us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:54:48.029864 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:54:48.029866 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:54:48.029867 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:54:48.029932 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:54:48.030133 (+ 201us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:54:48.030135 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:54:48.030136 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:54:48.030272 (+ 136us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:54:48.030363 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0927 14:54:48.030550 (+ 187us) server_negotiation.cc:299] Negotiation successful
0927 14:54:48.030635 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260927 14:54:48.033037 16313 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57682:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 14:54:48.036192 16313 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 14:54:48.043996 16435 tablet_service.cc:1467] Processing CreateTablet for tablet fac237341d5e44a8bc772a814114074d (DEFAULT_TABLE table=test-table [id=e1fe7f09be0246be93fecf923c725a5e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 14:54:48.044590 16435 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fac237341d5e44a8bc772a814114074d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 14:54:48.047521 16393 tablet_bootstrap.cc:492] T fac237341d5e44a8bc772a814114074d P f8ee67065e464c1b95de6b57be9697ad: Bootstrap starting.
I20260927 14:54:48.047642 16508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:54:48.042920 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:46285 (local address 127.0.0.1:42754)
0927 14:54:48.043044 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0927 14:54:48.043059 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 14:54:48.043139 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:54:48.044012 (+ 873us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:54:48.044015 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:54:48.044031 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 14:54:48.044129 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:54:48.044135 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:54:48.045429 (+ 1294us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:54:48.045433 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:54:48.046355 (+ 922us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:54:48.046362 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:54:48.047159 (+ 797us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:54:48.047162 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:54:48.047284 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:54:48.047334 (+ 50us) client_negotiation.cc:769] Sending connection context
0927 14:54:48.047434 (+ 100us) client_negotiation.cc:240] Negotiation successful
0927 14:54:48.047519 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":45}
I20260927 14:54:48.048455 16393 tablet_bootstrap.cc:654] T fac237341d5e44a8bc772a814114074d P f8ee67065e464c1b95de6b57be9697ad: Neither blocks nor log segments found. Creating new log.
I20260927 14:54:48.049485 16780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:54:48.043121 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42754 (local address 127.0.98.3:46285)
0927 14:54:48.043494 (+ 373us) server_negotiation.cc:206] Beginning negotiation
0927 14:54:48.043506 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 14:54:48.043531 (+ 25us) server_negotiation.cc:407] Connection header received
0927 14:54:48.043612 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:54:48.043619 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:54:48.043869 (+ 250us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:54:48.044037 (+ 168us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 14:54:48.044232 (+ 195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:54:48.045327 (+ 1095us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:54:48.046466 (+ 1139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:54:48.047034 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:54:48.047148 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:54:48.047291 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0927 14:54:48.049188 (+ 1897us) server_negotiation.cc:299] Negotiation successful
0927 14:54:48.049299 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":282,"thread_start_us":203,"threads_started":1}
I20260927 14:54:48.049603 16781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:54:48.043208 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:37931 (local address 127.0.0.1:39384)
0927 14:54:48.043592 (+ 384us) negotiation.cc:107] Waiting for socket to connect
0927 14:54:48.043605 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 14:54:48.043689 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:54:48.045836 (+ 2147us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:54:48.045839 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:54:48.045856 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 14:54:48.045935 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:54:48.045940 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:54:48.047252 (+ 1312us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:54:48.047256 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:54:48.048335 (+ 1079us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:54:48.048345 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:54:48.049218 (+ 873us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:54:48.049221 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:54:48.049304 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:54:48.049333 (+ 29us) client_negotiation.cc:769] Sending connection context
0927 14:54:48.049409 (+ 76us) client_negotiation.cc:240] Negotiation successful
0927 14:54:48.049487 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":324,"mutex_wait_us":1,"thread_start_us":105,"threads_started":1}
I20260927 14:54:48.049667 16782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:54:48.043625 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39384 (local address 127.0.98.2:37931)
0927 14:54:48.045397 (+ 1772us) server_negotiation.cc:206] Beginning negotiation
0927 14:54:48.045408 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 14:54:48.045430 (+ 22us) server_negotiation.cc:407] Connection header received
0927 14:54:48.045500 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:54:48.045504 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:54:48.045677 (+ 173us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:54:48.045847 (+ 170us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 14:54:48.046019 (+ 172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:54:48.047112 (+ 1093us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:54:48.048454 (+ 1342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:54:48.049099 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:54:48.049218 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:54:48.049358 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0927 14:54:48.049436 (+ 78us) server_negotiation.cc:299] Negotiation successful
0927 14:54:48.049535 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1650,"thread_start_us":95,"threads_started":1}
I20260927 14:54:48.049834 16393 tablet_bootstrap.cc:492] T fac237341d5e44a8bc772a814114074d P f8ee67065e464c1b95de6b57be9697ad: No bootstrap required, opened a new log
I20260927 14:54:48.049922 16393 ts_tablet_manager.cc:1397] T fac237341d5e44a8bc772a814114074d P f8ee67065e464c1b95de6b57be9697ad: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260927 14:54:48.050333 16393 raft_consensus.cc:348] T fac237341d5e44a8bc772a814114074d P f8ee67065e464c1b95de6b57be9697ad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9eab78e4338043749a6dbec46092f279" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46285 } } peers { permanent_uuid: "f3e6feeb2c794c849ab9ef7fbef5fb4c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37931 } } peers { permanent_uuid: "f8ee67065e464c1b95de6b57be9697ad" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34745 } }
I20260927 14:54:48.050442 16393 raft_consensus.cc:374] T fac237341d5e44a8bc772a814114074d P f8ee67065e464c1b95de6b57be9697ad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 14:54:48.050478 16393 raft_consensus.cc:729] T fac237341d5e44a8bc772a814114074d P f8ee67065e464c1b95de6b57be9697ad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f8ee67065e464c1b95de6b57be9697ad, State: Initialized, Role: FOLLOWER
I20260927 14:54:48.050596 16393 consensus_queue.cc:260] T fac237341d5e44a8bc772a814114074d P f8ee67065e464c1b95de6b57be9697ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9eab78e4338043749a6dbec46092f279" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46285 } } peers { permanent_uuid: "f3e6feeb2c794c849ab9ef7fbef5fb4c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37931 } } peers { permanent_uuid: "f8ee67065e464c1b95de6b57be9697ad" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34745 } }
I20260927 14:54:48.050881 16393 ts_tablet_manager.cc:1428] T fac237341d5e44a8bc772a814114074d P f8ee67065e464c1b95de6b57be9697ad: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260927 14:54:48.052575 16705 tablet_service.cc:1467] Processing CreateTablet for tablet fac237341d5e44a8bc772a814114074d (DEFAULT_TABLE table=test-table [id=e1fe7f09be0246be93fecf923c725a5e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 14:54:48.052575 16573 tablet_service.cc:1467] Processing CreateTablet for tablet fac237341d5e44a8bc772a814114074d (DEFAULT_TABLE table=test-table [id=e1fe7f09be0246be93fecf923c725a5e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 14:54:48.053671 16705 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fac237341d5e44a8bc772a814114074d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 14:54:48.053668 16573 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fac237341d5e44a8bc772a814114074d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 14:54:48.061461 16785 tablet_bootstrap.cc:492] T fac237341d5e44a8bc772a814114074d P f3e6feeb2c794c849ab9ef7fbef5fb4c: Bootstrap starting.
I20260927 14:54:48.061928 16784 tablet_bootstrap.cc:492] T fac237341d5e44a8bc772a814114074d P 9eab78e4338043749a6dbec46092f279: Bootstrap starting.
I20260927 14:54:48.064154 16784 tablet_bootstrap.cc:654] T fac237341d5e44a8bc772a814114074d P 9eab78e4338043749a6dbec46092f279: Neither blocks nor log segments found. Creating new log.
I20260927 14:54:48.064610 16785 tablet_bootstrap.cc:654] T fac237341d5e44a8bc772a814114074d P f3e6feeb2c794c849ab9ef7fbef5fb4c: Neither blocks nor log segments found. Creating new log.
I20260927 14:54:48.065495 16784 log.cc:824] T fac237341d5e44a8bc772a814114074d P 9eab78e4338043749a6dbec46092f279: Log is configured to *not* fsync() on all Append() calls
I20260927 14:54:48.065496 16785 log.cc:824] T fac237341d5e44a8bc772a814114074d P f3e6feeb2c794c849ab9ef7fbef5fb4c: Log is configured to *not* fsync() on all Append() calls
I20260927 14:54:48.067471 16785 tablet_bootstrap.cc:492] T fac237341d5e44a8bc772a814114074d P f3e6feeb2c794c849ab9ef7fbef5fb4c: No bootstrap required, opened a new log
I20260927 14:54:48.067471 16784 tablet_bootstrap.cc:492] T fac237341d5e44a8bc772a814114074d P 9eab78e4338043749a6dbec46092f279: No bootstrap required, opened a new log
I20260927 14:54:48.067660 16785 ts_tablet_manager.cc:1397] T fac237341d5e44a8bc772a814114074d P f3e6feeb2c794c849ab9ef7fbef5fb4c: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260927 14:54:48.067660 16784 ts_tablet_manager.cc:1397] T fac237341d5e44a8bc772a814114074d P 9eab78e4338043749a6dbec46092f279: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260927 14:54:48.071551 16784 raft_consensus.cc:348] T fac237341d5e44a8bc772a814114074d P 9eab78e4338043749a6dbec46092f279 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9eab78e4338043749a6dbec46092f279" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46285 } } peers { permanent_uuid: "f3e6feeb2c794c849ab9ef7fbef5fb4c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37931 } } peers { permanent_uuid: "f8ee67065e464c1b95de6b57be9697ad" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34745 } }
I20260927 14:54:48.071551 16785 raft_consensus.cc:348] T fac237341d5e44a8bc772a814114074d P f3e6feeb2c794c849ab9ef7fbef5fb4c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9eab78e4338043749a6dbec46092f279" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46285 } } peers { permanent_uuid: "f3e6feeb2c794c849ab9ef7fbef5fb4c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37931 } } peers { permanent_uuid: "f8ee67065e464c1b95de6b57be9697ad" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34745 } }
I20260927 14:54:48.071831 16785 raft_consensus.cc:374] T fac237341d5e44a8bc772a814114074d P f3e6feeb2c794c849ab9ef7fbef5fb4c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 14:54:48.071830 16784 raft_consensus.cc:374] T fac237341d5e44a8bc772a814114074d P 9eab78e4338043749a6dbec46092f279 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 14:54:48.071894 16785 raft_consensus.cc:729] T fac237341d5e44a8bc772a814114074d P f3e6feeb2c794c849ab9ef7fbef5fb4c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f3e6feeb2c794c849ab9ef7fbef5fb4c, State: Initialized, Role: FOLLOWER
I20260927 14:54:48.071900 16784 raft_consensus.cc:729] T fac237341d5e44a8bc772a814114074d P 9eab78e4338043749a6dbec46092f279 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9eab78e4338043749a6dbec46092f279, State: Initialized, Role: FOLLOWER
I20260927 14:54:48.072347 16784 consensus_queue.cc:260] T fac237341d5e44a8bc772a814114074d P 9eab78e4338043749a6dbec46092f279 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9eab78e4338043749a6dbec46092f279" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46285 } } peers { permanent_uuid: "f3e6feeb2c794c849ab9ef7fbef5fb4c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37931 } } peers { permanent_uuid: "f8ee67065e464c1b95de6b57be9697ad" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34745 } }
I20260927 14:54:48.072347 16785 consensus_queue.cc:260] T fac237341d5e44a8bc772a814114074d P f3e6feeb2c794c849ab9ef7fbef5fb4c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9eab78e4338043749a6dbec46092f279" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46285 } } peers { permanent_uuid: "f3e6feeb2c794c849ab9ef7fbef5fb4c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37931 } } peers { permanent_uuid: "f8ee67065e464c1b95de6b57be9697ad" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34745 } }
I20260927 14:54:48.072976 16639 heartbeater.cc:503] Master 127.0.98.62:34779 was elected leader, sending a full tablet report...
I20260927 14:54:48.072976 16771 heartbeater.cc:503] Master 127.0.98.62:34779 was elected leader, sending a full tablet report...
I20260927 14:54:48.073362 16784 ts_tablet_manager.cc:1428] T fac237341d5e44a8bc772a814114074d P 9eab78e4338043749a6dbec46092f279: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260927 14:54:48.073362 16785 ts_tablet_manager.cc:1428] T fac237341d5e44a8bc772a814114074d P f3e6feeb2c794c849ab9ef7fbef5fb4c: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
W20260927 14:54:48.093870 16502 tablet.cc:2369] T fac237341d5e44a8bc772a814114074d P f8ee67065e464c1b95de6b57be9697ad: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 14:54:48.250114 16772 tablet.cc:2369] T fac237341d5e44a8bc772a814114074d P 9eab78e4338043749a6dbec46092f279: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 14:54:48.259042 16789 raft_consensus.cc:482] T fac237341d5e44a8bc772a814114074d P f3e6feeb2c794c849ab9ef7fbef5fb4c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 14:54:48.259256 16789 raft_consensus.cc:504] T fac237341d5e44a8bc772a814114074d P f3e6feeb2c794c849ab9ef7fbef5fb4c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9eab78e4338043749a6dbec46092f279" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46285 } } peers { permanent_uuid: "f3e6feeb2c794c849ab9ef7fbef5fb4c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37931 } } peers { permanent_uuid: "f8ee67065e464c1b95de6b57be9697ad" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34745 } }
I20260927 14:54:48.260450 16789 leader_election.cc:288] T fac237341d5e44a8bc772a814114074d P f3e6feeb2c794c849ab9ef7fbef5fb4c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9eab78e4338043749a6dbec46092f279 (127.0.98.3:46285), f8ee67065e464c1b95de6b57be9697ad (127.0.98.1:34745)
I20260927 14:54:48.264441 16641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:54:48.260655 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:46285 (local address 127.0.98.2:59681)
0927 14:54:48.260860 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0927 14:54:48.260875 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 14:54:48.260974 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:54:48.261342 (+ 368us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:54:48.261345 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:54:48.261365 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 14:54:48.261514 (+ 149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:54:48.261525 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:54:48.262329 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:54:48.262332 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:54:48.263239 (+ 907us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:54:48.263248 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:54:48.263953 (+ 705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:54:48.263957 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:54:48.264075 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:54:48.264113 (+ 38us) client_negotiation.cc:769] Sending connection context
0927 14:54:48.264184 (+ 71us) client_negotiation.cc:240] Negotiation successful
0927 14:54:48.264255 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":106,"mutex_wait_us":3}
I20260927 14:54:48.264456 16780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:54:48.260758 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:59681 (local address 127.0.98.3:46285)
0927 14:54:48.260905 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0927 14:54:48.260911 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 14:54:48.260928 (+ 17us) server_negotiation.cc:407] Connection header received
0927 14:54:48.261131 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:54:48.261135 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:54:48.261202 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:54:48.261284 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 14:54:48.261625 (+ 341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:54:48.262219 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:54:48.263380 (+ 1161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:54:48.263832 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:54:48.263921 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:54:48.263970 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0927 14:54:48.264217 (+ 247us) server_negotiation.cc:299] Negotiation successful
0927 14:54:48.264296 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260927 14:54:48.264622 16790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:54:48.260729 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:34745 (local address 127.0.98.2:45455)
0927 14:54:48.261144 (+ 415us) negotiation.cc:107] Waiting for socket to connect
0927 14:54:48.261157 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 14:54:48.261265 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:54:48.261610 (+ 345us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:54:48.261613 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:54:48.261630 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 14:54:48.261746 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:54:48.261752 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:54:48.262552 (+ 800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:54:48.262559 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:54:48.263410 (+ 851us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:54:48.263420 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:54:48.264141 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:54:48.264143 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:54:48.264305 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:54:48.264330 (+ 25us) client_negotiation.cc:769] Sending connection context
0927 14:54:48.264406 (+ 76us) client_negotiation.cc:240] Negotiation successful
0927 14:54:48.264471 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":350,"thread_start_us":99,"threads_started":1}
I20260927 14:54:48.264662 16791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:54:48.260819 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:45455 (local address 127.0.98.1:34745)
0927 14:54:48.261128 (+ 309us) server_negotiation.cc:206] Beginning negotiation
0927 14:54:48.261132 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 14:54:48.261242 (+ 110us) server_negotiation.cc:407] Connection header received
0927 14:54:48.261387 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:54:48.261390 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:54:48.261453 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:54:48.261574 (+ 121us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 14:54:48.261854 (+ 280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:54:48.262397 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:54:48.263526 (+ 1129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:54:48.264031 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:54:48.264133 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:54:48.264298 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0927 14:54:48.264432 (+ 134us) server_negotiation.cc:299] Negotiation successful
0927 14:54:48.264514 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":197,"thread_start_us":85,"threads_started":1}
I20260927 14:54:48.265270 16455 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fac237341d5e44a8bc772a814114074d" candidate_uuid: "f3e6feeb2c794c849ab9ef7fbef5fb4c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f8ee67065e464c1b95de6b57be9697ad" is_pre_election: true
I20260927 14:54:48.265347 16725 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fac237341d5e44a8bc772a814114074d" candidate_uuid: "f3e6feeb2c794c849ab9ef7fbef5fb4c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9eab78e4338043749a6dbec46092f279" is_pre_election: true
I20260927 14:54:48.265611 16455 raft_consensus.cc:2454] T fac237341d5e44a8bc772a814114074d P f8ee67065e464c1b95de6b57be9697ad [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f3e6feeb2c794c849ab9ef7fbef5fb4c in term 0.
I20260927 14:54:48.265627 16725 raft_consensus.cc:2454] T fac237341d5e44a8bc772a814114074d P 9eab78e4338043749a6dbec46092f279 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f3e6feeb2c794c849ab9ef7fbef5fb4c in term 0.
I20260927 14:54:48.266121 16527 leader_election.cc:302] T fac237341d5e44a8bc772a814114074d P f3e6feeb2c794c849ab9ef7fbef5fb4c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 9eab78e4338043749a6dbec46092f279, f3e6feeb2c794c849ab9ef7fbef5fb4c, f8ee67065e464c1b95de6b57be9697ad; no voters:
I20260927 14:54:48.266386 16789 raft_consensus.cc:2781] T fac237341d5e44a8bc772a814114074d P f3e6feeb2c794c849ab9ef7fbef5fb4c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 14:54:48.266469 16789 raft_consensus.cc:482] T fac237341d5e44a8bc772a814114074d P f3e6feeb2c794c849ab9ef7fbef5fb4c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 14:54:48.266511 16789 raft_consensus.cc:3037] T fac237341d5e44a8bc772a814114074d P f3e6feeb2c794c849ab9ef7fbef5fb4c [term 0 FOLLOWER]: Advancing to term 1
I20260927 14:54:48.267501 16789 raft_consensus.cc:504] T fac237341d5e44a8bc772a814114074d P f3e6feeb2c794c849ab9ef7fbef5fb4c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9eab78e4338043749a6dbec46092f279" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46285 } } peers { permanent_uuid: "f3e6feeb2c794c849ab9ef7fbef5fb4c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37931 } } peers { permanent_uuid: "f8ee67065e464c1b95de6b57be9697ad" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34745 } }
I20260927 14:54:48.267827 16789 leader_election.cc:288] T fac237341d5e44a8bc772a814114074d P f3e6feeb2c794c849ab9ef7fbef5fb4c [CANDIDATE]: Term 1 election: Requested vote from peers 9eab78e4338043749a6dbec46092f279 (127.0.98.3:46285), f8ee67065e464c1b95de6b57be9697ad (127.0.98.1:34745)
I20260927 14:54:48.268106 16725 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fac237341d5e44a8bc772a814114074d" candidate_uuid: "f3e6feeb2c794c849ab9ef7fbef5fb4c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9eab78e4338043749a6dbec46092f279"
I20260927 14:54:48.268132 16455 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fac237341d5e44a8bc772a814114074d" candidate_uuid: "f3e6feeb2c794c849ab9ef7fbef5fb4c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f8ee67065e464c1b95de6b57be9697ad"
I20260927 14:54:48.268230 16725 raft_consensus.cc:3037] T fac237341d5e44a8bc772a814114074d P 9eab78e4338043749a6dbec46092f279 [term 0 FOLLOWER]: Advancing to term 1
I20260927 14:54:48.268239 16455 raft_consensus.cc:3037] T fac237341d5e44a8bc772a814114074d P f8ee67065e464c1b95de6b57be9697ad [term 0 FOLLOWER]: Advancing to term 1
I20260927 14:54:48.269518 16725 raft_consensus.cc:2454] T fac237341d5e44a8bc772a814114074d P 9eab78e4338043749a6dbec46092f279 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f3e6feeb2c794c849ab9ef7fbef5fb4c in term 1.
I20260927 14:54:48.269518 16455 raft_consensus.cc:2454] T fac237341d5e44a8bc772a814114074d P f8ee67065e464c1b95de6b57be9697ad [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f3e6feeb2c794c849ab9ef7fbef5fb4c in term 1.
I20260927 14:54:48.269821 16527 leader_election.cc:302] T fac237341d5e44a8bc772a814114074d P f3e6feeb2c794c849ab9ef7fbef5fb4c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: f3e6feeb2c794c849ab9ef7fbef5fb4c, f8ee67065e464c1b95de6b57be9697ad; no voters:
I20260927 14:54:48.270025 16789 raft_consensus.cc:2781] T fac237341d5e44a8bc772a814114074d P f3e6feeb2c794c849ab9ef7fbef5fb4c [term 1 FOLLOWER]: Leader election won for term 1
I20260927 14:54:48.270282 16789 raft_consensus.cc:686] T fac237341d5e44a8bc772a814114074d P f3e6feeb2c794c849ab9ef7fbef5fb4c [term 1 LEADER]: Becoming Leader. State: Replica: f3e6feeb2c794c849ab9ef7fbef5fb4c, State: Running, Role: LEADER
I20260927 14:54:48.270640 16789 consensus_queue.cc:237] T fac237341d5e44a8bc772a814114074d P f3e6feeb2c794c849ab9ef7fbef5fb4c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9eab78e4338043749a6dbec46092f279" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46285 } } peers { permanent_uuid: "f3e6feeb2c794c849ab9ef7fbef5fb4c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37931 } } peers { permanent_uuid: "f8ee67065e464c1b95de6b57be9697ad" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34745 } }
I20260927 14:54:48.273689 16312 catalog_manager.cc:5495] T fac237341d5e44a8bc772a814114074d P f3e6feeb2c794c849ab9ef7fbef5fb4c reported cstate change: term changed from 0 to 1, leader changed from <none> to f3e6feeb2c794c849ab9ef7fbef5fb4c (127.0.98.2). New cstate: current_term: 1 leader_uuid: "f3e6feeb2c794c849ab9ef7fbef5fb4c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9eab78e4338043749a6dbec46092f279" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46285 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f3e6feeb2c794c849ab9ef7fbef5fb4c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37931 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f8ee67065e464c1b95de6b57be9697ad" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34745 } health_report { overall_health: UNKNOWN } } }
W20260927 14:54:48.304863 16640 tablet.cc:2369] T fac237341d5e44a8bc772a814114074d P f3e6feeb2c794c849ab9ef7fbef5fb4c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 14:54:48.339570 16782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:54:48.335983 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39396 (local address 127.0.98.2:37931)
0927 14:54:48.336127 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0927 14:54:48.336131 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 14:54:48.336147 (+ 16us) server_negotiation.cc:407] Connection header received
0927 14:54:48.336258 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:54:48.336261 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:54:48.336330 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:54:48.336446 (+ 116us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 14:54:48.336680 (+ 234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:54:48.337238 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:54:48.338076 (+ 838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:54:48.338299 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:54:48.338418 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:54:48.338727 (+ 309us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 14:54:48.339156 (+ 429us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 14:54:48.339295 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0927 14:54:48.339338 (+ 43us) server_negotiation.cc:299] Negotiation successful
0927 14:54:48.339396 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260927 14:54:48.345481 16725 raft_consensus.cc:1261] T fac237341d5e44a8bc772a814114074d P 9eab78e4338043749a6dbec46092f279 [term 1 FOLLOWER]: Refusing update from remote peer f3e6feeb2c794c849ab9ef7fbef5fb4c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 14:54:48.345479 16455 raft_consensus.cc:1261] T fac237341d5e44a8bc772a814114074d P f8ee67065e464c1b95de6b57be9697ad [term 1 FOLLOWER]: Refusing update from remote peer f3e6feeb2c794c849ab9ef7fbef5fb4c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 14:54:48.346161 16792 consensus_queue.cc:1035] T fac237341d5e44a8bc772a814114074d P f3e6feeb2c794c849ab9ef7fbef5fb4c [LEADER]: Connected to new peer: Peer: permanent_uuid: "f8ee67065e464c1b95de6b57be9697ad" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34745 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 14:54:48.346406 16789 consensus_queue.cc:1035] T fac237341d5e44a8bc772a814114074d P f3e6feeb2c794c849ab9ef7fbef5fb4c [LEADER]: Connected to new peer: Peer: permanent_uuid: "9eab78e4338043749a6dbec46092f279" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 14:54:48.354120 16798 mvcc.cc:205] Tried to move back new op lower bound from 7333973558655438848 to 7333973558359220224. Current Snapshot: MvccSnapshot[applied={T|T < 7333973558655438848}]
I20260927 14:54:48.361853 16313 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57682:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 14:54:48.362053 16313 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57682:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 14:54:48.364645 16313 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a992827594d14e428ab1ee64535827e5: Sending DeleteTablet for 3 replicas of tablet fac237341d5e44a8bc772a814114074d
I20260927 14:54:48.365299 16435 tablet_service.cc:1514] Processing DeleteTablet for tablet fac237341d5e44a8bc772a814114074d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 14:54:48 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:43598
I20260927 14:54:48.365350 16705 tablet_service.cc:1514] Processing DeleteTablet for tablet fac237341d5e44a8bc772a814114074d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 14:54:48 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:42754
I20260927 14:54:48.365842 16805 tablet_replica.cc:330] T fac237341d5e44a8bc772a814114074d P f8ee67065e464c1b95de6b57be9697ad: stopping tablet replica
I20260927 14:54:48.365973 16805 raft_consensus.cc:2229] T fac237341d5e44a8bc772a814114074d P f8ee67065e464c1b95de6b57be9697ad [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 14:54:48.366101 16805 raft_consensus.cc:2258] T fac237341d5e44a8bc772a814114074d P f8ee67065e464c1b95de6b57be9697ad [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 14:54:48.366077 16573 tablet_service.cc:1514] Processing DeleteTablet for tablet fac237341d5e44a8bc772a814114074d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 14:54:48 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:39384
I20260927 14:54:48.367017 16805 ts_tablet_manager.cc:1905] T fac237341d5e44a8bc772a814114074d P f8ee67065e464c1b95de6b57be9697ad: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 14:54:48.367031 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6936m5/build/debug/bin/kudu with pid 16351
I20260927 14:54:48.367766 16806 tablet_replica.cc:330] T fac237341d5e44a8bc772a814114074d P 9eab78e4338043749a6dbec46092f279: stopping tablet replica
I20260927 14:54:48.368083 16806 raft_consensus.cc:2229] T fac237341d5e44a8bc772a814114074d P 9eab78e4338043749a6dbec46092f279 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 14:54:48.368417 16806 raft_consensus.cc:2258] T fac237341d5e44a8bc772a814114074d P 9eab78e4338043749a6dbec46092f279 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 14:54:48.370051 16806 ts_tablet_manager.cc:1905] T fac237341d5e44a8bc772a814114074d P 9eab78e4338043749a6dbec46092f279: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 14:54:48.372718 16806 ts_tablet_manager.cc:1918] T fac237341d5e44a8bc772a814114074d P 9eab78e4338043749a6dbec46092f279: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 14:54:48.372850 16806 log.cc:1196] T fac237341d5e44a8bc772a814114074d P 9eab78e4338043749a6dbec46092f279: Deleting WAL directory at /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790520684602251-392-0/minicluster-data/ts-2/wal/wals/fac237341d5e44a8bc772a814114074d
I20260927 14:54:48.373250 16806 ts_tablet_manager.cc:1939] T fac237341d5e44a8bc772a814114074d P 9eab78e4338043749a6dbec46092f279: Deleting consensus metadata
I20260927 14:54:48.374549 16299 catalog_manager.cc:4841] TS 9eab78e4338043749a6dbec46092f279 (127.0.98.3:46285): tablet fac237341d5e44a8bc772a814114074d (table test-table [id=e1fe7f09be0246be93fecf923c725a5e]) successfully deleted
W20260927 14:54:48.375684 16298 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:34745 (error 108)
W20260927 14:54:48.377141 16781 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 14:54:48.376471 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:34745 (local address 127.0.0.1:43612)
0927 14:54:48.376600 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0927 14:54:48.376617 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 14:54:48.376704 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:54:48.377026 (+ 322us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:34745: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":50}
I20260927 14:54:48.377316 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6936m5/build/debug/bin/kudu with pid 16505
W20260927 14:54:48.377353 16298 catalog_manager.cc:4584] TS f8ee67065e464c1b95de6b57be9697ad (127.0.98.1:34745): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet fac237341d5e44a8bc772a814114074d: Network error: Client connection negotiation failed: client connection to 127.0.98.1:34745: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260927 14:54:48.383370 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6936m5/build/debug/bin/kudu with pid 16642
I20260927 14:54:48.383613 16781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:54:48.383289 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:37931 (local address 127.0.0.1:39412)
0927 14:54:48.383419 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0927 14:54:48.383520 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:37931: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260927 14:54:48.388968 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6936m5/build/debug/bin/kudu with pid 16280
I20260927 14:55:08.395602 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6936m5/build/debug/bin/kudu
/tmp/dist-test-task6936m5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790520684602251-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790520684602251-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790520684602251-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790520684602251-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34779
--webserver_interface=127.0.98.62
--webserver_port=44315
--keytab_file=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790520684602251-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40871
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:34779
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790520684602251-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790520684602251-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790520684602251-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 14:55:08.505825 16809 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 14:55:08.506074 16809 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 14:55:08.506112 16809 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 14:55:08.509989 16809 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260927 14:55:08.510037 16809 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260927 14:55:08.510052 16809 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260927 14:55:08.510066 16809 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 14:55:08.510083 16809 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 14:55:08.510102 16809 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 14:55:08.510118 16809 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 14:55:08.510131 16809 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 14:55:08.515121 16809 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40871
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790520684602251-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790520684602251-392-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:34779
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790520684602251-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34779
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790520684602251-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=44315
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16809
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790520684602251-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:15:17 UTC on ccycloud.quasar-ledxfe.root.comops.site
build id 4676134
I20260927 14:55:08.516005 16809 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 14:55:08.516887 16809 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 14:55:08.525455 16817 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 14:55:08.525478 16815 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 14:55:08.525492 16809 server_base.cc:1034] running on GCE node
W20260927 14:55:08.525540 16814 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 14:55:08.526259 16809 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 14:55:08.527263 16809 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 14:55:08.528420 16809 hybrid_clock.cc:648] HybridClock initialized: now 1790520908528394 us; error 39 us; skew 500 ppm
Sep 27 14:55:08 dist-test-slave-xk1g krb5kdc[15732](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790520908, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 14:55:08.535431 16809 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 14:55:08.536334 16809 webserver.cc:460] Webserver started at http://127.0.98.62:44315/ using document root <none> and password file <none>
I20260927 14:55:08.536882 16809 fs_manager.cc:359] Metadata directory not provided
I20260927 14:55:08.536974 16809 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 14:55:08.540632 16809 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260927 14:55:08.542763 16824 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 14:55:08.543916 16809 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 14:55:08.544077 16809 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790520684602251-392-0/minicluster-data/master-0/data,/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790520684602251-392-0/minicluster-data/master-0/wal
uuid: "a992827594d14e428ab1ee64535827e5"
format_stamp: "Formatted at 2026-09-27 14:54:26 on dist-test-slave-xk1g"
server_key: "d50aee429206bc58e096762fd7248ed8"
server_key_iv: "82418a00305e96599705060c600e3e83"
server_key_version: "encryptionkey@0"
I20260927 14:55:08.544596 16809 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790520684602251-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790520684602251-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790520684602251-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 14:55:08.575716 16809 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 14:55:08.578794 16809 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 14:55:08.579015 16809 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 14:55:08.586369 16809 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:34779
I20260927 14:55:08.586678 16876 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:34779 every 8 connection(s)
I20260927 14:55:08.587533 16809 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790520684602251-392-0/minicluster-data/master-0/data/info.pb
I20260927 14:55:08.591812 16877 sys_catalog.cc:263] Verifying existing consensus state
I20260927 14:55:08.592026 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6936m5/build/debug/bin/kudu as pid 16809
I20260927 14:55:08.592432 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6936m5/build/debug/bin/kudu
/tmp/dist-test-task6936m5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790520684602251-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790520684602251-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790520684602251-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790520684602251-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:34745
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:34779
--webserver_port=40025
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790520684602251-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40871
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790520684602251-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790520684602251-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790520684602251-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 14:55:08.593611 16877 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a992827594d14e428ab1ee64535827e5: Bootstrap starting.
I20260927 14:55:08.611044 16877 log.cc:824] T 00000000000000000000000000000000 P a992827594d14e428ab1ee64535827e5: Log is configured to *not* fsync() on all Append() calls
I20260927 14:55:08.622848 16877 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a992827594d14e428ab1ee64535827e5: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 14:55:08.623238 16877 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a992827594d14e428ab1ee64535827e5: Bootstrap complete.
I20260927 14:55:08.627259 16877 raft_consensus.cc:348] T 00000000000000000000000000000000 P a992827594d14e428ab1ee64535827e5 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a992827594d14e428ab1ee64535827e5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34779 } }
I20260927 14:55:08.627662 16877 raft_consensus.cc:729] T 00000000000000000000000000000000 P a992827594d14e428ab1ee64535827e5 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: a992827594d14e428ab1ee64535827e5, State: Initialized, Role: FOLLOWER
I20260927 14:55:08.628145 16877 consensus_queue.cc:260] T 00000000000000000000000000000000 P a992827594d14e428ab1ee64535827e5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a992827594d14e428ab1ee64535827e5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34779 } }
I20260927 14:55:08.628301 16877 raft_consensus.cc:388] T 00000000000000000000000000000000 P a992827594d14e428ab1ee64535827e5 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 14:55:08.628398 16877 raft_consensus.cc:482] T 00000000000000000000000000000000 P a992827594d14e428ab1ee64535827e5 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 14:55:08.628525 16877 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a992827594d14e428ab1ee64535827e5 [term 2 FOLLOWER]: Advancing to term 3
I20260927 14:55:08.630458 16877 raft_consensus.cc:504] T 00000000000000000000000000000000 P a992827594d14e428ab1ee64535827e5 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a992827594d14e428ab1ee64535827e5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34779 } }
I20260927 14:55:08.630788 16877 leader_election.cc:302] T 00000000000000000000000000000000 P a992827594d14e428ab1ee64535827e5 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a992827594d14e428ab1ee64535827e5; no voters:
I20260927 14:55:08.631116 16877 leader_election.cc:288] T 00000000000000000000000000000000 P a992827594d14e428ab1ee64535827e5 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260927 14:55:08.631246 16881 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a992827594d14e428ab1ee64535827e5 [term 3 FOLLOWER]: Leader election won for term 3
I20260927 14:55:08.631479 16881 raft_consensus.cc:686] T 00000000000000000000000000000000 P a992827594d14e428ab1ee64535827e5 [term 3 LEADER]: Becoming Leader. State: Replica: a992827594d14e428ab1ee64535827e5, State: Running, Role: LEADER
I20260927 14:55:08.631809 16881 consensus_queue.cc:237] T 00000000000000000000000000000000 P a992827594d14e428ab1ee64535827e5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a992827594d14e428ab1ee64535827e5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34779 } }
I20260927 14:55:08.632351 16877 sys_catalog.cc:564] T 00000000000000000000000000000000 P a992827594d14e428ab1ee64535827e5 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 14:55:08.632725 16882 sys_catalog.cc:455] T 00000000000000000000000000000000 P a992827594d14e428ab1ee64535827e5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "a992827594d14e428ab1ee64535827e5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a992827594d14e428ab1ee64535827e5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34779 } } }
I20260927 14:55:08.632838 16882 sys_catalog.cc:458] T 00000000000000000000000000000000 P a992827594d14e428ab1ee64535827e5 [sys.catalog]: This master's current role is: LEADER
I20260927 14:55:08.633052 16883 sys_catalog.cc:455] T 00000000000000000000000000000000 P a992827594d14e428ab1ee64535827e5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a992827594d14e428ab1ee64535827e5. Latest consensus state: current_term: 3 leader_uuid: "a992827594d14e428ab1ee64535827e5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a992827594d14e428ab1ee64535827e5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34779 } } }
I20260927 14:55:08.633159 16883 sys_catalog.cc:458] T 00000000000000000000000000000000 P a992827594d14e428ab1ee64535827e5 [sys.catalog]: This master's current role is: LEADER
I20260927 14:55:08.633440 16885 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 14:55:08.638727 16885 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 14:55:08.639447 16885 catalog_manager.cc:1259] Loaded cluster ID: d488d1bfd5274fd3ab35889a54cbc5d0
I20260927 14:55:08.639518 16885 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 14:55:08.641868 16885 catalog_manager.cc:1504] Loading token signing keys...
I20260927 14:55:08.642544 16885 catalog_manager.cc:5879] T 00000000000000000000000000000000 P a992827594d14e428ab1ee64535827e5: Loaded TSK: 0
I20260927 14:55:08.642623 16885 catalog_manager.cc:5879] T 00000000000000000000000000000000 P a992827594d14e428ab1ee64535827e5: Loaded TSK: 1
F20260927 14:55:08.643747 16885 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7fd1634cd9ad google::LogMessage::Fail() at ??:0
@ 0x7fd1634d088a google::LogMessage::SendToLog() at ??:0
@ 0x7fd1634cd4de google::LogMessage::Flush() at ??:0
@ 0x7fd1634cea19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7fd16619118d kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7fd166189f0d kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7fd16de327bb kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7fd16de0c99c kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7fd16de0d178 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlvE3_clEv at ??:0
@ 0x7fd16de4edf8 _ZNSt17_Function_handlerIFN4kudu6StatusEvEZNS0_6master14CatalogManager24PrepareForLeadershipTaskEvEUlvE3_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x7fd16de6d82b std::function<>::operator()() at ??:0
@ 0x7fd16de0cb4f _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_ at ??:0
@ 0x7fd16de0df3e kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7fd16de0a579 _ZZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvENKUlvE_clEv at ??:0
@ 0x7fd16de4e640 _ZNSt17_Function_handlerIFvvEZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7fd16571f3d7 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7fd16571fc83 _ZZN4kudu10ThreadPool12CreateThreadEvENKUlvE_clEv at ??:0
@ 0x7fd1657213f2 _ZNSt17_Function_handlerIFvvEZN4kudu10ThreadPool12CreateThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7fd165710512 kudu::Thread::SuperviseThread() at ??:0
@ 0x7fd166d546db start_thread at ??:0
@ 0x7fd161fcb61f clone at ??:0
W20260927 14:55:08.745172 16879 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 14:55:08.745556 16879 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 14:55:08.745654 16879 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 14:55:08.752074 16879 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 14:55:08.752131 16879 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 14:55:08.752213 16879 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 14:55:08.759527 16879 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40871
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790520684602251-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790520684602251-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790520684602251-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:34745
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790520684602251-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=40025
--tserver_master_addrs=127.0.98.62:34779
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16879
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790520684602251-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:15:17 UTC on ccycloud.quasar-ledxfe.root.comops.site
build id 4676134
I20260927 14:55:08.760813 16879 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 14:55:08.762085 16879 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 14:55:08.774541 16927 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 14:55:08.774544 16924 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 14:55:08.774541 16925 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 14:55:08.774734 16879 server_base.cc:1034] running on GCE node
I20260927 14:55:08.775386 16879 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 14:55:08.776139 16879 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 14:55:08.777324 16879 hybrid_clock.cc:648] HybridClock initialized: now 1790520908777299 us; error 35 us; skew 500 ppm
Sep 27 14:55:08 dist-test-slave-xk1g krb5kdc[15732](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790520908, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 14:55:08.780539 16879 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 14:55:08.781507 16879 webserver.cc:460] Webserver started at http://127.0.98.1:40025/ using document root <none> and password file <none>
I20260927 14:55:08.782068 16879 fs_manager.cc:359] Metadata directory not provided
I20260927 14:55:08.782133 16879 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 14:55:08.786006 16879 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260927 14:55:08.788265 16934 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 14:55:08.789640 16879 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 14:55:08.789768 16879 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790520684602251-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790520684602251-392-0/minicluster-data/ts-0/wal
uuid: "f8ee67065e464c1b95de6b57be9697ad"
format_stamp: "Formatted at 2026-09-27 14:54:26 on dist-test-slave-xk1g"
server_key: "952ee1f62b1ebc8e55e8c976234b4dce"
server_key_iv: "592e8d010d72b7b41a99653c5bc2d2e6"
server_key_version: "encryptionkey@0"
I20260927 14:55:08.790290 16879 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790520684602251-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790520684602251-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790520684602251-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 14:55:08.807289 16879 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 14:55:08.811007 16879 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 14:55:08.811209 16879 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 14:55:08.811923 16879 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 14:55:08.813293 16941 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260927 14:55:08.817399 16879 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260927 14:55:08.817471 16879 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.004s user 0.000s sys 0.000s
I20260927 14:55:08.817533 16879 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260927 14:55:08.819731 16879 ts_tablet_manager.cc:610] Registered 1 tablets
I20260927 14:55:08.819803 16879 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260927 14:55:08.820087 16941 tablet_bootstrap.cc:492] T fac237341d5e44a8bc772a814114074d P f8ee67065e464c1b95de6b57be9697ad: Bootstrap starting.
I20260927 14:55:08.826283 16941 log.cc:824] T fac237341d5e44a8bc772a814114074d P f8ee67065e464c1b95de6b57be9697ad: Log is configured to *not* fsync() on all Append() calls
I20260927 14:55:08.830657 16879 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:34745
I20260927 14:55:08.830672 17049 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:34745 every 8 connection(s)
I20260927 14:55:08.831727 16879 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790520684602251-392-0/minicluster-data/ts-0/data/info.pb
I20260927 14:55:08.834280 16941 tablet_bootstrap.cc:492] T fac237341d5e44a8bc772a814114074d P f8ee67065e464c1b95de6b57be9697ad: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 14:55:08.834704 16941 tablet_bootstrap.cc:492] T fac237341d5e44a8bc772a814114074d P f8ee67065e464c1b95de6b57be9697ad: Bootstrap complete.
I20260927 14:55:08.835070 16941 ts_tablet_manager.cc:1397] T fac237341d5e44a8bc772a814114074d P f8ee67065e464c1b95de6b57be9697ad: Time spent bootstrapping tablet: real 0.015s user 0.014s sys 0.000s
I20260927 14:55:08.837508 16941 raft_consensus.cc:348] T fac237341d5e44a8bc772a814114074d P f8ee67065e464c1b95de6b57be9697ad [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9eab78e4338043749a6dbec46092f279" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46285 } } peers { permanent_uuid: "f3e6feeb2c794c849ab9ef7fbef5fb4c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37931 } } peers { permanent_uuid: "f8ee67065e464c1b95de6b57be9697ad" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34745 } }
I20260927 14:55:08.837925 16941 raft_consensus.cc:729] T fac237341d5e44a8bc772a814114074d P f8ee67065e464c1b95de6b57be9697ad [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: f8ee67065e464c1b95de6b57be9697ad, State: Initialized, Role: FOLLOWER
I20260927 14:55:08.838409 16941 consensus_queue.cc:260] T fac237341d5e44a8bc772a814114074d P f8ee67065e464c1b95de6b57be9697ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9eab78e4338043749a6dbec46092f279" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46285 } } peers { permanent_uuid: "f3e6feeb2c794c849ab9ef7fbef5fb4c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37931 } } peers { permanent_uuid: "f8ee67065e464c1b95de6b57be9697ad" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34745 } }
I20260927 14:55:08.839190 16941 ts_tablet_manager.cc:1428] T fac237341d5e44a8bc772a814114074d P f8ee67065e464c1b95de6b57be9697ad: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20260927 14:55:08.839491 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6936m5/build/debug/bin/kudu as pid 16879
I20260927 14:55:08.839926 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6936m5/build/debug/bin/kudu
/tmp/dist-test-task6936m5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790520684602251-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790520684602251-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790520684602251-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790520684602251-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:37931
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:34779
--webserver_port=33621
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790520684602251-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40871
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790520684602251-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790520684602251-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790520684602251-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 14:55:08.918668 17052 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 14:55:08.834148 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34779 (local address 127.0.98.1:41455)
0927 14:55:08.834692 (+ 544us) negotiation.cc:107] Waiting for socket to connect
0927 14:55:08.834748 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0927 14:55:08.835819 (+ 1071us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:55:08.917829 (+ 82010us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34779: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":321,"thread_start_us":132,"threads_started":1}
W20260927 14:55:08.919181 17050 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:34779 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:34779: Client connection negotiation failed: client connection to 127.0.98.62:34779: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260927 14:55:08.919974 17052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:55:08.919709 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34779 (local address 127.0.98.1:53667)
0927 14:55:08.919845 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0927 14:55:08.919915 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34779: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260927 14:55:08.920657 17052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:55:08.920449 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34779 (local address 127.0.98.1:55433)
0927 14:55:08.920554 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0927 14:55:08.920610 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34779: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260927 14:55:08.920849 17050 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260927 14:55:08.974984 17054 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 14:55:08.975279 17054 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 14:55:08.975356 17054 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 14:55:08.979413 17054 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 14:55:08.979504 17054 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 14:55:08.979598 17054 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 14:55:08.984716 17054 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40871
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790520684602251-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790520684602251-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790520684602251-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:37931
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790520684602251-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=33621
--tserver_master_addrs=127.0.98.62:34779
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17054
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790520684602251-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:15:17 UTC on ccycloud.quasar-ledxfe.root.comops.site
build id 4676134
I20260927 14:55:08.985759 17054 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 14:55:08.986807 17054 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 14:55:08.996465 17060 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 14:55:08.996481 17059 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 14:55:08.996513 17054 server_base.cc:1034] running on GCE node
W20260927 14:55:08.996472 17062 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 14:55:08.997218 17054 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 14:55:08.997797 17054 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 14:55:08.998961 17054 hybrid_clock.cc:648] HybridClock initialized: now 1790520908998928 us; error 44 us; skew 500 ppm
Sep 27 14:55:09 dist-test-slave-xk1g krb5kdc[15732](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790520909, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 14:55:09.001715 17054 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 14:55:09.002622 17054 webserver.cc:460] Webserver started at http://127.0.98.2:33621/ using document root <none> and password file <none>
I20260927 14:55:09.003187 17054 fs_manager.cc:359] Metadata directory not provided
I20260927 14:55:09.003275 17054 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 14:55:09.007192 17054 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 14:55:09.009414 17069 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 14:55:09.010639 17054 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260927 14:55:09.010778 17054 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790520684602251-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790520684602251-392-0/minicluster-data/ts-1/wal
uuid: "f3e6feeb2c794c849ab9ef7fbef5fb4c"
format_stamp: "Formatted at 2026-09-27 14:54:26 on dist-test-slave-xk1g"
server_key: "d4aafd3368d18eac59a1a1b0b80dca96"
server_key_iv: "bd37185f913e046786f09f6fba5e5470"
server_key_version: "encryptionkey@0"
I20260927 14:55:09.011340 17054 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790520684602251-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790520684602251-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790520684602251-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 14:55:09.033209 17054 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 14:55:09.036072 17054 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 14:55:09.036278 17054 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 14:55:09.036926 17054 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 14:55:09.038084 17076 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260927 14:55:09.042384 17054 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260927 14:55:09.042487 17054 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.005s user 0.000s sys 0.000s
I20260927 14:55:09.042580 17054 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260927 14:55:09.044268 17054 ts_tablet_manager.cc:610] Registered 1 tablets
I20260927 14:55:09.044342 17054 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260927 14:55:09.044586 17076 tablet_bootstrap.cc:492] T fac237341d5e44a8bc772a814114074d P f3e6feeb2c794c849ab9ef7fbef5fb4c: Bootstrap starting.
I20260927 14:55:09.054999 17054 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:37931
I20260927 14:55:09.055086 17183 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:37931 every 8 connection(s)
I20260927 14:55:09.055990 17054 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790520684602251-392-0/minicluster-data/ts-1/data/info.pb
I20260927 14:55:09.057351 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6936m5/build/debug/bin/kudu as pid 17054
I20260927 14:55:09.057833 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6936m5/build/debug/bin/kudu
/tmp/dist-test-task6936m5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790520684602251-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790520684602251-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790520684602251-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790520684602251-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:46285
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:34779
--webserver_port=45735
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790520684602251-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40871
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790520684602251-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790520684602251-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790520684602251-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 14:55:09.059604 17186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:55:09.058048 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34779 (local address 127.0.98.2:46761)
0927 14:55:09.058562 (+ 514us) negotiation.cc:107] Waiting for socket to connect
0927 14:55:09.058694 (+ 132us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34779: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":291,"thread_start_us":142,"threads_started":1}
W20260927 14:55:09.060175 17184 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:34779 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:34779: Client connection negotiation failed: client connection to 127.0.98.62:34779: connect: Connection refused (error 111)
I20260927 14:55:09.060895 17186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:55:09.060612 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34779 (local address 127.0.98.2:60753)
0927 14:55:09.060754 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0927 14:55:09.060810 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34779: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260927 14:55:09.061604 17186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:55:09.061370 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34779 (local address 127.0.98.2:35895)
0927 14:55:09.061493 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0927 14:55:09.061543 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34779: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260927 14:55:09.061771 17184 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260927 14:55:09.063236 17076 log.cc:824] T fac237341d5e44a8bc772a814114074d P f3e6feeb2c794c849ab9ef7fbef5fb4c: Log is configured to *not* fsync() on all Append() calls
I20260927 14:55:09.070227 17076 tablet_bootstrap.cc:492] T fac237341d5e44a8bc772a814114074d P f3e6feeb2c794c849ab9ef7fbef5fb4c: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 14:55:09.070591 17076 tablet_bootstrap.cc:492] T fac237341d5e44a8bc772a814114074d P f3e6feeb2c794c849ab9ef7fbef5fb4c: Bootstrap complete.
I20260927 14:55:09.071700 17076 ts_tablet_manager.cc:1397] T fac237341d5e44a8bc772a814114074d P f3e6feeb2c794c849ab9ef7fbef5fb4c: Time spent bootstrapping tablet: real 0.027s user 0.016s sys 0.008s
I20260927 14:55:09.074440 17076 raft_consensus.cc:348] T fac237341d5e44a8bc772a814114074d P f3e6feeb2c794c849ab9ef7fbef5fb4c [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9eab78e4338043749a6dbec46092f279" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46285 } } peers { permanent_uuid: "f3e6feeb2c794c849ab9ef7fbef5fb4c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37931 } } peers { permanent_uuid: "f8ee67065e464c1b95de6b57be9697ad" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34745 } }
I20260927 14:55:09.074970 17076 raft_consensus.cc:729] T fac237341d5e44a8bc772a814114074d P f3e6feeb2c794c849ab9ef7fbef5fb4c [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: f3e6feeb2c794c849ab9ef7fbef5fb4c, State: Initialized, Role: FOLLOWER
I20260927 14:55:09.075421 17076 consensus_queue.cc:260] T fac237341d5e44a8bc772a814114074d P f3e6feeb2c794c849ab9ef7fbef5fb4c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9eab78e4338043749a6dbec46092f279" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46285 } } peers { permanent_uuid: "f3e6feeb2c794c849ab9ef7fbef5fb4c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37931 } } peers { permanent_uuid: "f8ee67065e464c1b95de6b57be9697ad" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34745 } }
I20260927 14:55:09.076439 17076 ts_tablet_manager.cc:1428] T fac237341d5e44a8bc772a814114074d P f3e6feeb2c794c849ab9ef7fbef5fb4c: Time spent starting tablet: real 0.005s user 0.007s sys 0.000s
I20260927 14:55:09.076649 17186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:55:09.076365 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34779 (local address 127.0.98.2:34441)
0927 14:55:09.076500 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0927 14:55:09.076570 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34779: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260927 14:55:09.077332 17186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:55:09.077123 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34779 (local address 127.0.98.2:56465)
0927 14:55:09.077225 (+ 102us) negotiation.cc:107] Waiting for socket to connect
0927 14:55:09.077275 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34779: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260927 14:55:09.170089 17187 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 14:55:09.170331 17187 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 14:55:09.170372 17187 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 14:55:09.174265 17187 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 14:55:09.174320 17187 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 14:55:09.174377 17187 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 14:55:09.179508 17187 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40871
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790520684602251-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790520684602251-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790520684602251-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:46285
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790520684602251-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=45735
--tserver_master_addrs=127.0.98.62:34779
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17187
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790520684602251-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:15:17 UTC on ccycloud.quasar-ledxfe.root.comops.site
build id 4676134
I20260927 14:55:09.180471 17187 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 14:55:09.181432 17187 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 14:55:09.190619 17194 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 14:55:09.190541 17195 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 14:55:09.190728 17187 server_base.cc:1034] running on GCE node
W20260927 14:55:09.190569 17197 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 14:55:09.191226 17187 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 14:55:09.191847 17187 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 14:55:09.193070 17187 hybrid_clock.cc:648] HybridClock initialized: now 1790520909193023 us; error 41 us; skew 500 ppm
Sep 27 14:55:09 dist-test-slave-xk1g krb5kdc[15732](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790520909, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 14:55:09.195793 17187 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 14:55:09.196661 17187 webserver.cc:460] Webserver started at http://127.0.98.3:45735/ using document root <none> and password file <none>
I20260927 14:55:09.197271 17187 fs_manager.cc:359] Metadata directory not provided
I20260927 14:55:09.197366 17187 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 14:55:09.200907 17187 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260927 14:55:09.202987 17204 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 14:55:09.204140 17187 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 14:55:09.204270 17187 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790520684602251-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790520684602251-392-0/minicluster-data/ts-2/wal
uuid: "9eab78e4338043749a6dbec46092f279"
format_stamp: "Formatted at 2026-09-27 14:54:26 on dist-test-slave-xk1g"
server_key: "678491584feb82d497710ade28548c61"
server_key_iv: "acf9c8bba7ec8bb4f65321f6ec1f109c"
server_key_version: "encryptionkey@0"
I20260927 14:55:09.204785 17187 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790520684602251-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790520684602251-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790520684602251-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 14:55:09.220206 17187 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 14:55:09.223038 17187 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 14:55:09.223245 17187 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 14:55:09.223809 17187 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 14:55:09.224632 17187 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 14:55:09.224704 17187 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 14:55:09.224776 17187 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 14:55:09.224829 17187 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 14:55:09.233533 17187 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:46285
I20260927 14:55:09.233564 17317 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:46285 every 8 connection(s)
I20260927 14:55:09.234493 17187 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6936m5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790520684602251-392-0/minicluster-data/ts-2/data/info.pb
I20260927 14:55:09.237347 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6936m5/build/debug/bin/kudu as pid 17187
I20260927 14:55:09.238610 17320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:55:09.236589 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34779 (local address 127.0.98.3:52847)
0927 14:55:09.237124 (+ 535us) negotiation.cc:107] Waiting for socket to connect
0927 14:55:09.237253 (+ 129us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34779: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":269,"thread_start_us":103,"threads_started":1}
W20260927 14:55:09.239038 17318 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:34779 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:34779: Client connection negotiation failed: client connection to 127.0.98.62:34779: connect: Connection refused (error 111)
I20260927 14:55:09.239590 17320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:55:09.239388 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34779 (local address 127.0.98.3:32917)
0927 14:55:09.239494 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0927 14:55:09.239537 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34779: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20260927 14:55:09.240129 17320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:55:09.239946 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34779 (local address 127.0.98.3:51453)
0927 14:55:09.240044 (+ 98us) negotiation.cc:107] Waiting for socket to connect
0927 14:55:09.240084 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34779: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260927 14:55:09.240298 17318 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260927 14:55:09.921859 17322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:55:09.921432 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34779 (local address 127.0.98.1:52005)
0927 14:55:09.921711 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0927 14:55:09.921763 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34779: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":69,"threads_started":1}
I20260927 14:55:10.078580 17323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:55:10.078159 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34779 (local address 127.0.98.2:52487)
0927 14:55:10.078423 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0927 14:55:10.078484 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34779: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":73,"threads_started":1}
I20260927 14:55:10.241307 17324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:55:10.240819 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34779 (local address 127.0.98.3:51679)
0927 14:55:10.241147 (+ 328us) negotiation.cc:107] Waiting for socket to connect
0927 14:55:10.241212 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34779: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":74,"threads_started":1}
I20260927 14:55:10.280436 17325 raft_consensus.cc:482] T fac237341d5e44a8bc772a814114074d P f8ee67065e464c1b95de6b57be9697ad [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 14:55:10.280619 17325 raft_consensus.cc:504] T fac237341d5e44a8bc772a814114074d P f8ee67065e464c1b95de6b57be9697ad [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9eab78e4338043749a6dbec46092f279" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46285 } } peers { permanent_uuid: "f3e6feeb2c794c849ab9ef7fbef5fb4c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37931 } } peers { permanent_uuid: "f8ee67065e464c1b95de6b57be9697ad" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34745 } }
I20260927 14:55:10.281710 17325 leader_election.cc:288] T fac237341d5e44a8bc772a814114074d P f8ee67065e464c1b95de6b57be9697ad [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9eab78e4338043749a6dbec46092f279 (127.0.98.3:46285), f3e6feeb2c794c849ab9ef7fbef5fb4c (127.0.98.2:37931)
Sep 27 14:55:10 dist-test-slave-xk1g krb5kdc[15732](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790520908, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.3@KRBTEST.COM
Sep 27 14:55:10 dist-test-slave-xk1g krb5kdc[15732](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790520908, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.2@KRBTEST.COM
I20260927 14:55:10.294823 17322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:55:10.281914 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:46285 (local address 127.0.98.1:39955)
0927 14:55:10.282140 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0927 14:55:10.282155 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 14:55:10.282262 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:55:10.283742 (+ 1480us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:55:10.283745 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:55:10.284021 (+ 276us) client_negotiation.cc:189] Negotiated authn=SASL
0927 14:55:10.284540 (+ 519us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:55:10.284553 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:55:10.285857 (+ 1304us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:55:10.285860 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:55:10.286718 (+ 858us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:55:10.286727 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:55:10.287411 (+ 684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:55:10.287415 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:55:10.287641 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:55:10.288477 (+ 836us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 14:55:10.288488 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 14:55:10.290302 (+ 1814us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 14:55:10.292336 (+ 2034us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 14:55:10.292341 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 14:55:10.292353 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0927 14:55:10.292591 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 14:55:10.292921 (+ 330us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 14:55:10.292924 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 14:55:10.292926 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 14:55:10.293147 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 14:55:10.293798 (+ 651us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 14:55:10.293804 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 14:55:10.294060 (+ 256us) client_negotiation.cc:769] Sending connection context
0927 14:55:10.294262 (+ 202us) client_negotiation.cc:240] Negotiation successful
0927 14:55:10.294565 (+ 303us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":151,"mutex_wait_us":154,"spinlock_wait_cycles":14976}
I20260927 14:55:10.294823 17327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:55:10.282106 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:39955 (local address 127.0.98.3:46285)
0927 14:55:10.282510 (+ 404us) server_negotiation.cc:206] Beginning negotiation
0927 14:55:10.282530 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0927 14:55:10.282565 (+ 35us) server_negotiation.cc:407] Connection header received
0927 14:55:10.283224 (+ 659us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:55:10.283229 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:55:10.283506 (+ 277us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:55:10.283821 (+ 315us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:55:10.284708 (+ 887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:55:10.285701 (+ 993us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:55:10.286848 (+ 1147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:55:10.287287 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:55:10.287470 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:55:10.290506 (+ 3036us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:55:10.290511 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:55:10.290522 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 14:55:10.290548 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:55:10.292220 (+ 1672us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:55:10.292706 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:55:10.292716 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:55:10.292720 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:55:10.292821 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:55:10.293274 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:55:10.293278 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:55:10.293280 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:55:10.293659 (+ 379us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:55:10.293824 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0927 14:55:10.294372 (+ 548us) server_negotiation.cc:299] Negotiation successful
0927 14:55:10.294636 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":284,"thread_start_us":84,"threads_started":1}
I20260927 14:55:10.294888 17328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:55:10.282132 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:47479 (local address 127.0.98.2:37931)
0927 14:55:10.282558 (+ 426us) server_negotiation.cc:206] Beginning negotiation
0927 14:55:10.282567 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 14:55:10.282598 (+ 31us) server_negotiation.cc:407] Connection header received
0927 14:55:10.282805 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:55:10.282809 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:55:10.283117 (+ 308us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:55:10.283423 (+ 306us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:55:10.284700 (+ 1277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:55:10.285717 (+ 1017us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:55:10.286842 (+ 1125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:55:10.287290 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:55:10.287466 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:55:10.290809 (+ 3343us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:55:10.290817 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:55:10.290836 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 14:55:10.290877 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:55:10.292626 (+ 1749us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:55:10.293155 (+ 529us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:55:10.293160 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:55:10.293164 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:55:10.293325 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:55:10.293615 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:55:10.293618 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:55:10.293619 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:55:10.293989 (+ 370us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:55:10.294125 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0927 14:55:10.294448 (+ 323us) server_negotiation.cc:299] Negotiation successful
0927 14:55:10.294731 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":315,"thread_start_us":101,"threads_started":1}
I20260927 14:55:10.294874 17326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:55:10.281956 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:37931 (local address 127.0.98.1:47479)
0927 14:55:10.282295 (+ 339us) negotiation.cc:107] Waiting for socket to connect
0927 14:55:10.282310 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 14:55:10.282409 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:55:10.283452 (+ 1043us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:55:10.283458 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:55:10.284015 (+ 557us) client_negotiation.cc:189] Negotiated authn=SASL
0927 14:55:10.284539 (+ 524us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:55:10.284553 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:55:10.285864 (+ 1311us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:55:10.285867 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:55:10.286720 (+ 853us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:55:10.286729 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:55:10.287410 (+ 681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:55:10.287414 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:55:10.287645 (+ 231us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:55:10.288433 (+ 788us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 14:55:10.288457 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0927 14:55:10.290645 (+ 2188us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 14:55:10.292758 (+ 2113us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 14:55:10.292761 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 14:55:10.292763 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 14:55:10.292958 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 14:55:10.293430 (+ 472us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 14:55:10.293434 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 14:55:10.293436 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 14:55:10.293521 (+ 85us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 14:55:10.294095 (+ 574us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 14:55:10.294098 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 14:55:10.294192 (+ 94us) client_negotiation.cc:769] Sending connection context
0927 14:55:10.294301 (+ 109us) client_negotiation.cc:240] Negotiation successful
0927 14:55:10.294658 (+ 357us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":279,"mutex_wait_us":226,"spinlock_wait_cycles":11648,"thread_start_us":76,"threads_started":1}
I20260927 14:55:10.296854 17138 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fac237341d5e44a8bc772a814114074d" candidate_uuid: "f8ee67065e464c1b95de6b57be9697ad" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "f3e6feeb2c794c849ab9ef7fbef5fb4c" is_pre_election: true
I20260927 14:55:10.297201 17138 raft_consensus.cc:2454] T fac237341d5e44a8bc772a814114074d P f3e6feeb2c794c849ab9ef7fbef5fb4c [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f8ee67065e464c1b95de6b57be9697ad in term 1.
I20260927 14:55:10.298120 16938 leader_election.cc:302] T fac237341d5e44a8bc772a814114074d P f8ee67065e464c1b95de6b57be9697ad [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: f3e6feeb2c794c849ab9ef7fbef5fb4c, f8ee67065e464c1b95de6b57be9697ad; no voters:
W20260927 14:55:10.298326 16937 leader_election.cc:341] T fac237341d5e44a8bc772a814114074d P f8ee67065e464c1b95de6b57be9697ad [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 9eab78e4338043749a6dbec46092f279 (127.0.98.3:46285): Not found: Tablet not found: fac237341d5e44a8bc772a814114074d
I20260927 14:55:10.298465 17325 raft_consensus.cc:2781] T fac237341d5e44a8bc772a814114074d P f8ee67065e464c1b95de6b57be9697ad [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260927 14:55:10.298578 17325 raft_consensus.cc:482] T fac237341d5e44a8bc772a814114074d P f8ee67065e464c1b95de6b57be9697ad [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 14:55:10.298635 17325 raft_consensus.cc:3037] T fac237341d5e44a8bc772a814114074d P f8ee67065e464c1b95de6b57be9697ad [term 1 FOLLOWER]: Advancing to term 2
I20260927 14:55:10.300906 17325 raft_consensus.cc:504] T fac237341d5e44a8bc772a814114074d P f8ee67065e464c1b95de6b57be9697ad [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9eab78e4338043749a6dbec46092f279" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46285 } } peers { permanent_uuid: "f3e6feeb2c794c849ab9ef7fbef5fb4c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37931 } } peers { permanent_uuid: "f8ee67065e464c1b95de6b57be9697ad" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34745 } }
I20260927 14:55:10.301278 17325 leader_election.cc:288] T fac237341d5e44a8bc772a814114074d P f8ee67065e464c1b95de6b57be9697ad [CANDIDATE]: Term 2 election: Requested vote from peers 9eab78e4338043749a6dbec46092f279 (127.0.98.3:46285), f3e6feeb2c794c849ab9ef7fbef5fb4c (127.0.98.2:37931)
I20260927 14:55:10.301684 17138 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fac237341d5e44a8bc772a814114074d" candidate_uuid: "f8ee67065e464c1b95de6b57be9697ad" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "f3e6feeb2c794c849ab9ef7fbef5fb4c"
I20260927 14:55:10.301815 17138 raft_consensus.cc:3037] T fac237341d5e44a8bc772a814114074d P f3e6feeb2c794c849ab9ef7fbef5fb4c [term 1 FOLLOWER]: Advancing to term 2
W20260927 14:55:10.301925 16937 leader_election.cc:341] T fac237341d5e44a8bc772a814114074d P f8ee67065e464c1b95de6b57be9697ad [CANDIDATE]: Term 2 election: Tablet error from VoteRequest() call to peer 9eab78e4338043749a6dbec46092f279 (127.0.98.3:46285): Not found: Tablet not found: fac237341d5e44a8bc772a814114074d
I20260927 14:55:10.303634 17138 raft_consensus.cc:2454] T fac237341d5e44a8bc772a814114074d P f3e6feeb2c794c849ab9ef7fbef5fb4c [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f8ee67065e464c1b95de6b57be9697ad in term 2.
I20260927 14:55:10.303930 16938 leader_election.cc:302] T fac237341d5e44a8bc772a814114074d P f8ee67065e464c1b95de6b57be9697ad [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: f3e6feeb2c794c849ab9ef7fbef5fb4c, f8ee67065e464c1b95de6b57be9697ad; no voters: 9eab78e4338043749a6dbec46092f279
I20260927 14:55:10.304102 17325 raft_consensus.cc:2781] T fac237341d5e44a8bc772a814114074d P f8ee67065e464c1b95de6b57be9697ad [term 2 FOLLOWER]: Leader election won for term 2
I20260927 14:55:10.304277 17325 raft_consensus.cc:686] T fac237341d5e44a8bc772a814114074d P f8ee67065e464c1b95de6b57be9697ad [term 2 LEADER]: Becoming Leader. State: Replica: f8ee67065e464c1b95de6b57be9697ad, State: Running, Role: LEADER
I20260927 14:55:10.304598 17325 consensus_queue.cc:237] T fac237341d5e44a8bc772a814114074d P f8ee67065e464c1b95de6b57be9697ad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9eab78e4338043749a6dbec46092f279" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46285 } } peers { permanent_uuid: "f3e6feeb2c794c849ab9ef7fbef5fb4c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37931 } } peers { permanent_uuid: "f8ee67065e464c1b95de6b57be9697ad" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34745 } }
I20260927 14:55:10.304937 17326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:55:10.304699 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34779 (local address 127.0.98.1:53713)
0927 14:55:10.304810 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0927 14:55:10.304863 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34779: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260927 14:55:10.750988 16937 consensus_peers.cc:487] T fac237341d5e44a8bc772a814114074d P f8ee67065e464c1b95de6b57be9697ad -> Peer 9eab78e4338043749a6dbec46092f279 (127.0.98.3:46285): Couldn't send request to peer 9eab78e4338043749a6dbec46092f279. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: fac237341d5e44a8bc772a814114074d. This is attempt 1: this message will repeat every 5th retry.
I20260927 14:55:10.899271 17138 raft_consensus.cc:1261] T fac237341d5e44a8bc772a814114074d P f3e6feeb2c794c849ab9ef7fbef5fb4c [term 2 FOLLOWER]: Refusing update from remote peer f8ee67065e464c1b95de6b57be9697ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20260927 14:55:10.900028 17325 consensus_queue.cc:1035] T fac237341d5e44a8bc772a814114074d P f8ee67065e464c1b95de6b57be9697ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "f3e6feeb2c794c849ab9ef7fbef5fb4c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37931 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260927 14:55:10.902164 17332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:55:10.901738 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34779 (local address 127.0.98.2:41027)
0927 14:55:10.901979 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0927 14:55:10.902043 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34779: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":55,"threads_started":1}
I20260927 14:55:10.904599 17332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:55:10.904382 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34779 (local address 127.0.98.2:60373)
0927 14:55:10.904499 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0927 14:55:10.904544 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34779: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260927 14:55:10.904610 17337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:55:10.904206 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34779 (local address 127.0.98.1:56437)
0927 14:55:10.904467 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0927 14:55:10.904528 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34779: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":53,"threads_started":1}
I20260927 14:55:11.242545 17338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:55:11.242087 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34779 (local address 127.0.98.3:60323)
0927 14:55:11.242367 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0927 14:55:11.242443 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34779: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":205,"thread_start_us":92,"threads_started":1}
I20260927 14:55:11.289337 17339 ts_tablet_manager.cc:927] T fac237341d5e44a8bc772a814114074d P 9eab78e4338043749a6dbec46092f279: Initiating tablet copy from peer f8ee67065e464c1b95de6b57be9697ad (127.0.98.1:34745)
I20260927 14:55:11.289965 17339 tablet_copy_client.cc:316] T fac237341d5e44a8bc772a814114074d P 9eab78e4338043749a6dbec46092f279: tablet copy: Beginning tablet copy session from remote peer at address 127.0.98.1:34745
Sep 27 14:55:11 dist-test-slave-xk1g krb5kdc[15732](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790520909, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260927 14:55:11.299687 17338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:55:11.290466 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:34745 (local address 127.0.98.3:53065)
0927 14:55:11.290597 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0927 14:55:11.290614 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 14:55:11.290730 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:55:11.291332 (+ 602us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:55:11.291338 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:55:11.291725 (+ 387us) client_negotiation.cc:189] Negotiated authn=SASL
0927 14:55:11.291846 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:55:11.291852 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:55:11.292827 (+ 975us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:55:11.292832 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:55:11.293602 (+ 770us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:55:11.293611 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:55:11.294203 (+ 592us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:55:11.294206 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:55:11.294311 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:55:11.294827 (+ 516us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 14:55:11.294838 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 14:55:11.296164 (+ 1326us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 14:55:11.297904 (+ 1740us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 14:55:11.297913 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 14:55:11.297921 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0927 14:55:11.298141 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 14:55:11.298438 (+ 297us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 14:55:11.298441 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 14:55:11.298443 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 14:55:11.298524 (+ 81us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 14:55:11.299065 (+ 541us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 14:55:11.299071 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 14:55:11.299198 (+ 127us) client_negotiation.cc:769] Sending connection context
0927 14:55:11.299337 (+ 139us) client_negotiation.cc:240] Negotiation successful
0927 14:55:11.299516 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":50}
I20260927 14:55:11.299731 17341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:55:11.290636 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:53065 (local address 127.0.98.1:34745)
0927 14:55:11.290935 (+ 299us) server_negotiation.cc:206] Beginning negotiation
0927 14:55:11.290946 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 14:55:11.290965 (+ 19us) server_negotiation.cc:407] Connection header received
0927 14:55:11.291006 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:55:11.291011 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:55:11.291181 (+ 170us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:55:11.291332 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:55:11.291976 (+ 644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:55:11.292713 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:55:11.293728 (+ 1015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:55:11.294087 (+ 359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:55:11.294203 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:55:11.296302 (+ 2099us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:55:11.296315 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:55:11.296325 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 14:55:11.296346 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:55:11.297793 (+ 1447us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:55:11.298253 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:55:11.298266 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:55:11.298267 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:55:11.298340 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:55:11.298615 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:55:11.298620 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:55:11.298622 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:55:11.298945 (+ 323us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:55:11.299094 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0927 14:55:11.299431 (+ 337us) server_negotiation.cc:299] Negotiation successful
0927 14:55:11.299571 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":188,"thread_start_us":83,"threads_started":1}
I20260927 14:55:11.300732 17023 tablet_copy_service.cc:140] P f8ee67065e464c1b95de6b57be9697ad: Received BeginTabletCopySession request for tablet fac237341d5e44a8bc772a814114074d from peer 9eab78e4338043749a6dbec46092f279 ({username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:53065)
I20260927 14:55:11.300899 17023 tablet_copy_service.cc:161] P f8ee67065e464c1b95de6b57be9697ad: Beginning new tablet copy session on tablet fac237341d5e44a8bc772a814114074d from peer 9eab78e4338043749a6dbec46092f279 at {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:53065: session id = 9eab78e4338043749a6dbec46092f279-fac237341d5e44a8bc772a814114074d
I20260927 14:55:11.302139 17023 tablet_copy_source_session.cc:205] T fac237341d5e44a8bc772a814114074d P f8ee67065e464c1b95de6b57be9697ad: Tablet Copy: opened 0 blocks and 1 log segments
I20260927 14:55:11.305917 17339 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fac237341d5e44a8bc772a814114074d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 14:55:11.312186 17339 tablet_copy_client.cc:791] T fac237341d5e44a8bc772a814114074d P 9eab78e4338043749a6dbec46092f279: tablet copy: Starting download of 0 data blocks...
I20260927 14:55:11.312376 17339 tablet_copy_client.cc:655] T fac237341d5e44a8bc772a814114074d P 9eab78e4338043749a6dbec46092f279: tablet copy: Starting download of 1 WAL segments...
I20260927 14:55:11.314005 17339 tablet_copy_client.cc:531] T fac237341d5e44a8bc772a814114074d P 9eab78e4338043749a6dbec46092f279: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260927 14:55:11.315644 17339 tablet_bootstrap.cc:492] T fac237341d5e44a8bc772a814114074d P 9eab78e4338043749a6dbec46092f279: Bootstrap starting.
I20260927 14:55:11.320639 17339 log.cc:824] T fac237341d5e44a8bc772a814114074d P 9eab78e4338043749a6dbec46092f279: Log is configured to *not* fsync() on all Append() calls
I20260927 14:55:11.326491 17339 tablet_bootstrap.cc:492] T fac237341d5e44a8bc772a814114074d P 9eab78e4338043749a6dbec46092f279: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 14:55:11.326773 17339 tablet_bootstrap.cc:492] T fac237341d5e44a8bc772a814114074d P 9eab78e4338043749a6dbec46092f279: Bootstrap complete.
I20260927 14:55:11.327064 17339 ts_tablet_manager.cc:1397] T fac237341d5e44a8bc772a814114074d P 9eab78e4338043749a6dbec46092f279: Time spent bootstrapping tablet: real 0.012s user 0.011s sys 0.000s
I20260927 14:55:11.329221 17339 raft_consensus.cc:348] T fac237341d5e44a8bc772a814114074d P 9eab78e4338043749a6dbec46092f279 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9eab78e4338043749a6dbec46092f279" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46285 } } peers { permanent_uuid: "f3e6feeb2c794c849ab9ef7fbef5fb4c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37931 } } peers { permanent_uuid: "f8ee67065e464c1b95de6b57be9697ad" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34745 } }
I20260927 14:55:11.329499 17339 raft_consensus.cc:729] T fac237341d5e44a8bc772a814114074d P 9eab78e4338043749a6dbec46092f279 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9eab78e4338043749a6dbec46092f279, State: Initialized, Role: FOLLOWER
I20260927 14:55:11.329941 17339 consensus_queue.cc:260] T fac237341d5e44a8bc772a814114074d P 9eab78e4338043749a6dbec46092f279 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 2.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9eab78e4338043749a6dbec46092f279" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46285 } } peers { permanent_uuid: "f3e6feeb2c794c849ab9ef7fbef5fb4c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37931 } } peers { permanent_uuid: "f8ee67065e464c1b95de6b57be9697ad" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34745 } }
I20260927 14:55:11.330812 17339 ts_tablet_manager.cc:1428] T fac237341d5e44a8bc772a814114074d P 9eab78e4338043749a6dbec46092f279: Time spent starting tablet: real 0.004s user 0.005s sys 0.000s
I20260927 14:55:11.331032 17338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:55:11.330783 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34779 (local address 127.0.98.3:36283)
0927 14:55:11.330912 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0927 14:55:11.330970 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34779: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260927 14:55:11.331530 17023 tablet_copy_service.cc:342] P f8ee67065e464c1b95de6b57be9697ad: Request end of tablet copy session 9eab78e4338043749a6dbec46092f279-fac237341d5e44a8bc772a814114074d received from {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:53065
I20260927 14:55:11.331633 17023 tablet_copy_service.cc:433] P f8ee67065e464c1b95de6b57be9697ad: ending tablet copy session 9eab78e4338043749a6dbec46092f279-fac237341d5e44a8bc772a814114074d on tablet fac237341d5e44a8bc772a814114074d with peer 9eab78e4338043749a6dbec46092f279
I20260927 14:55:11.331784 17338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:55:11.331524 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34779 (local address 127.0.98.3:58417)
0927 14:55:11.331671 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0927 14:55:11.331725 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34779: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
I20260927 14:55:11.768044 17345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:55:11.767603 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34779 (local address 127.0.98.1:37299)
0927 14:55:11.767868 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0927 14:55:11.767922 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34779: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":202,"thread_start_us":77,"threads_started":1}
I20260927 14:55:11.768046 17272 raft_consensus.cc:1203] T fac237341d5e44a8bc772a814114074d P 9eab78e4338043749a6dbec46092f279 [term 2 FOLLOWER]: Deduplicated request from leader. Original: 1.2->[2.3-2.3] Dedup: 2.3->[]
I20260927 14:55:11.769050 17338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:55:11.768745 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34779 (local address 127.0.98.3:38145)
0927 14:55:11.768907 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0927 14:55:11.768972 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34779: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
I20260927 14:55:11.905979 17346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:55:11.905535 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34779 (local address 127.0.98.2:49937)
0927 14:55:11.905791 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0927 14:55:11.905857 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34779: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":101,"threads_started":1}
I20260927 14:55:12.769212 17347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:55:12.768767 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34779 (local address 127.0.98.1:42563)
0927 14:55:12.769040 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0927 14:55:12.769116 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34779: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":72,"threads_started":1}
I20260927 14:55:12.770179 17348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:55:12.769763 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34779 (local address 127.0.98.3:45619)
0927 14:55:12.770037 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0927 14:55:12.770099 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34779: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":63,"threads_started":1}
I20260927 14:55:12.907224 17349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:55:12.906784 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34779 (local address 127.0.98.2:39773)
0927 14:55:12.907068 (+ 284us) negotiation.cc:107] Waiting for socket to connect
0927 14:55:12.907132 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34779: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":214,"thread_start_us":79,"threads_started":1}
I20260927 14:55:13.770442 17350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:55:13.770068 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34779 (local address 127.0.98.1:39391)
0927 14:55:13.770300 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0927 14:55:13.770350 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34779: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":168,"thread_start_us":74,"threads_started":1}
I20260927 14:55:13.771314 17351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:55:13.770983 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34779 (local address 127.0.98.3:36499)
0927 14:55:13.771189 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0927 14:55:13.771238 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34779: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":63,"threads_started":1}
I20260927 14:55:13.908422 17352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:55:13.908005 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34779 (local address 127.0.98.2:47357)
0927 14:55:13.908254 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0927 14:55:13.908309 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34779: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":88,"threads_started":1}
I20260927 14:55:14.771675 17353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:55:14.771269 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34779 (local address 127.0.98.1:34279)
0927 14:55:14.771507 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0927 14:55:14.771554 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34779: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":92,"threads_started":1}
I20260927 14:55:14.772428 17353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:55:14.772224 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34779 (local address 127.0.98.1:57965)
0927 14:55:14.772325 (+ 101us) negotiation.cc:107] Waiting for socket to connect
0927 14:55:14.772376 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34779: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
I20260927 14:55:14.772621 17354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:55:14.772194 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34779 (local address 127.0.98.3:37177)
0927 14:55:14.772445 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0927 14:55:14.772497 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34779: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":86,"threads_started":1}
I20260927 14:55:14.909792 17355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:55:14.909384 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34779 (local address 127.0.98.2:54963)
0927 14:55:14.909649 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0927 14:55:14.909699 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34779: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":101,"threads_started":1}
I20260927 14:55:15.773928 17357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:55:15.773477 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34779 (local address 127.0.98.3:58263)
0927 14:55:15.773743 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0927 14:55:15.773796 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34779: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":98,"threads_started":1}
I20260927 14:55:15.773977 17356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:55:15.773260 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34779 (local address 127.0.98.1:34937)
0927 14:55:15.773796 (+ 536us) negotiation.cc:107] Waiting for socket to connect
0927 14:55:15.773857 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34779: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":210,"thread_start_us":108,"threads_started":1}
I20260927 14:55:15.910913 17358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:55:15.910531 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34779 (local address 127.0.98.2:33449)
0927 14:55:15.910765 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0927 14:55:15.910819 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34779: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":166,"thread_start_us":75,"threads_started":1}
I20260927 14:55:16.775171 17361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:55:16.774744 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34779 (local address 127.0.98.1:42225)
0927 14:55:16.775023 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0927 14:55:16.775076 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34779: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":199,"thread_start_us":122,"threads_started":1}
I20260927 14:55:16.775166 17360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:55:16.774742 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34779 (local address 127.0.98.3:55843)
0927 14:55:16.775024 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0927 14:55:16.775075 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34779: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":209,"thread_start_us":119,"threads_started":1}
I20260927 14:55:16.912267 17362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:55:16.911776 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34779 (local address 127.0.98.2:49439)
0927 14:55:16.912076 (+ 300us) negotiation.cc:107] Waiting for socket to connect
0927 14:55:16.912138 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34779: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":221,"thread_start_us":109,"threads_started":1}
I20260927 14:55:17.776257 17363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:55:17.775806 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34779 (local address 127.0.98.1:39061)
0927 14:55:17.776064 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0927 14:55:17.776132 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34779: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":180,"thread_start_us":70,"threads_started":1}
I20260927 14:55:17.776472 17364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:55:17.776057 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34779 (local address 127.0.98.3:46211)
0927 14:55:17.776290 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0927 14:55:17.776365 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34779: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":169,"thread_start_us":81,"threads_started":1}
I20260927 14:55:17.913465 17365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:55:17.913026 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34779 (local address 127.0.98.2:56369)
0927 14:55:17.913321 (+ 295us) negotiation.cc:107] Waiting for socket to connect
0927 14:55:17.913371 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34779: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":230,"thread_start_us":93,"threads_started":1}
I20260927 14:55:18.777696 17366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:55:18.777183 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34779 (local address 127.0.98.1:40521)
0927 14:55:18.777515 (+ 332us) negotiation.cc:107] Waiting for socket to connect
0927 14:55:18.777583 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34779: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":243,"thread_start_us":98,"threads_started":1}
I20260927 14:55:18.777696 17367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:55:18.777234 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34779 (local address 127.0.98.3:54353)
0927 14:55:18.777515 (+ 281us) negotiation.cc:107] Waiting for socket to connect
0927 14:55:18.777583 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34779: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":204,"thread_start_us":67,"threads_started":1}
I20260927 14:55:18.914741 17368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:55:18.914232 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34779 (local address 127.0.98.2:52929)
0927 14:55:18.914512 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0927 14:55:18.914618 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34779: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":203,"thread_start_us":79,"threads_started":1}
I20260927 14:55:19.779042 17369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:55:19.778632 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34779 (local address 127.0.98.1:48759)
0927 14:55:19.778889 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0927 14:55:19.778945 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34779: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":173,"thread_start_us":97,"threads_started":1}
I20260927 14:55:19.779075 17370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:55:19.778630 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34779 (local address 127.0.98.3:58629)
0927 14:55:19.778942 (+ 312us) negotiation.cc:107] Waiting for socket to connect
0927 14:55:19.778991 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34779: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":249,"thread_start_us":99,"threads_started":1}
I20260927 14:55:19.916013 17371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:55:19.915602 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34779 (local address 127.0.98.2:59441)
0927 14:55:19.915861 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0927 14:55:19.915918 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34779: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":71,"threads_started":1}
I20260927 14:55:20.780331 17372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:55:20.779913 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34779 (local address 127.0.98.3:54227)
0927 14:55:20.780181 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0927 14:55:20.780237 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34779: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":94,"threads_started":1}
I20260927 14:55:20.780360 17373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:55:20.779961 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34779 (local address 127.0.98.1:53763)
0927 14:55:20.780214 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0927 14:55:20.780271 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34779: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":168,"thread_start_us":94,"threads_started":1}
I20260927 14:55:20.917253 17374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:55:20.916801 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34779 (local address 127.0.98.2:46785)
0927 14:55:20.917038 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0927 14:55:20.917158 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34779: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":166,"thread_start_us":67,"threads_started":1}
I20260927 14:55:21.781595 17375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:55:21.781175 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34779 (local address 127.0.98.3:56187)
0927 14:55:21.781423 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0927 14:55:21.781485 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34779: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":168,"thread_start_us":68,"threads_started":1}
I20260927 14:55:21.781687 17376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:55:21.781293 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34779 (local address 127.0.98.1:40939)
0927 14:55:21.781543 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0927 14:55:21.781605 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34779: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":76,"threads_started":1}
I20260927 14:55:21.918453 17377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:55:21.917974 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34779 (local address 127.0.98.2:43107)
0927 14:55:21.918243 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0927 14:55:21.918298 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34779: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":89,"threads_started":1}
I20260927 14:55:22.782898 17379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:55:22.782485 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34779 (local address 127.0.98.1:37343)
0927 14:55:22.782774 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0927 14:55:22.782818 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34779: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":236,"thread_start_us":79,"threads_started":1}
I20260927 14:55:22.782897 17378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:55:22.782454 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34779 (local address 127.0.98.3:39451)
0927 14:55:22.782746 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0927 14:55:22.782804 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34779: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":207,"thread_start_us":98,"threads_started":1}
I20260927 14:55:22.919663 17380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:55:22.919208 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34779 (local address 127.0.98.2:44885)
0927 14:55:22.919496 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0927 14:55:22.919560 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34779: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":74,"threads_started":1}
I20260927 14:55:23.784134 17381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:55:23.783744 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34779 (local address 127.0.98.3:35279)
0927 14:55:23.783997 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0927 14:55:23.784045 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34779: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":92,"threads_started":1}
I20260927 14:55:23.784155 17382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:55:23.783804 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34779 (local address 127.0.98.1:47177)
0927 14:55:23.784028 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0927 14:55:23.784076 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34779: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":83,"threads_started":1}
I20260927 14:55:23.920846 17383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:55:23.920446 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34779 (local address 127.0.98.2:46479)
0927 14:55:23.920697 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0927 14:55:23.920750 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34779: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":180,"thread_start_us":79,"threads_started":1}
I20260927 14:55:24.246372 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6936m5/build/debug/bin/kudu with pid 16879
I20260927 14:55:24.254457 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6936m5/build/debug/bin/kudu with pid 17054
I20260927 14:55:24.260013 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6936m5/build/debug/bin/kudu with pid 17187
2026-09-27T14:55:24Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (59316 ms)
[----------] 30 tests from SecurityITest (239652 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-task6936m5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790520684602251-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 14:55:24 dist-test-slave-xk1g krb5kdc[17386](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 14:55:24 dist-test-slave-xk1g krb5kdc[17386](info): set up 2 sockets
Sep 27 14:55:24 dist-test-slave-xk1g krb5kdc[17386](info): commencing operation
krb5kdc: starting...
W20260927 14:55:25.296788 392 mini_kdc.cc:121] Time spent starting KDC: real 1.014s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6936m5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790520684602251-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6936m5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790520684602251-392-0/krb5kdc/test-user.keytab.
Sep 27 14:55:25 dist-test-slave-xk1g krb5kdc[17386](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790520925, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T14:55:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T14:55:25Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6936m5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790520684602251-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6936m5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790520684602251-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 14:55:25.416317 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6936m5/build/debug/bin/kudu
/tmp/dist-test-task6936m5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790520684602251-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790520684602251-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790520684602251-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790520684602251-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40611
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790520684602251-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41785
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:40611
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790520684602251-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790520684602251-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790520684602251-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 14:55:25.526185 17399 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 14:55:25.526512 17399 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 14:55:25.526595 17399 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 14:55:25.530658 17399 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 14:55:25.530737 17399 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 14:55:25.530774 17399 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 14:55:25.530831 17399 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 14:55:25.530881 17399 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 14:55:25.536162 17399 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41785
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790520684602251-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790520684602251-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:40611
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790520684602251-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40611
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790520684602251-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17399
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790520684602251-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:15:17 UTC on ccycloud.quasar-ledxfe.root.comops.site
build id 4676134
I20260927 14:55:25.537357 17399 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 14:55:25.538415 17399 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 14:55:25.547025 17399 server_base.cc:1034] running on GCE node
W20260927 14:55:25.546988 17404 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 14:55:25.546993 17405 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 14:55:25.546988 17407 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 14:55:25.547760 17399 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 14:55:25.548804 17399 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 14:55:25.549961 17399 hybrid_clock.cc:648] HybridClock initialized: now 1790520925549944 us; error 34 us; skew 500 ppm
Sep 27 14:55:25 dist-test-slave-xk1g krb5kdc[17386](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790520925, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 14:55:25.552714 17399 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 14:55:25.553622 17399 webserver.cc:460] Webserver started at http://127.0.98.62:46275/ using document root <none> and password file <none>
I20260927 14:55:25.554121 17399 fs_manager.cc:359] Metadata directory not provided
I20260927 14:55:25.554175 17399 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 14:55:25.554343 17399 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 14:55:25.556262 17399 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6936m5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790520684602251-392-0/minicluster-data/master-0/data/instance:
uuid: "544504935a654d3d98f8d69af15791b9"
format_stamp: "Formatted at 2026-09-27 14:55:25 on dist-test-slave-xk1g"
server_key: "6044173de867c33f453fae081d6cd461"
server_key_iv: "73daa96c898f367cb50898ef1ef88895"
server_key_version: "encryptionkey@0"
I20260927 14:55:25.556749 17399 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6936m5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790520684602251-392-0/minicluster-data/master-0/wal/instance:
uuid: "544504935a654d3d98f8d69af15791b9"
format_stamp: "Formatted at 2026-09-27 14:55:25 on dist-test-slave-xk1g"
server_key: "6044173de867c33f453fae081d6cd461"
server_key_iv: "73daa96c898f367cb50898ef1ef88895"
server_key_version: "encryptionkey@0"
I20260927 14:55:25.560283 17399 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260927 14:55:25.562611 17414 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 14:55:25.563828 17399 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 14:55:25.563947 17399 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790520684602251-392-0/minicluster-data/master-0/data,/tmp/dist-test-task6936m5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790520684602251-392-0/minicluster-data/master-0/wal
uuid: "544504935a654d3d98f8d69af15791b9"
format_stamp: "Formatted at 2026-09-27 14:55:25 on dist-test-slave-xk1g"
server_key: "6044173de867c33f453fae081d6cd461"
server_key_iv: "73daa96c898f367cb50898ef1ef88895"
server_key_version: "encryptionkey@0"
I20260927 14:55:25.564046 17399 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790520684602251-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790520684602251-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790520684602251-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 14:55:25.580879 17399 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 14:55:25.583702 17399 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 14:55:25.583890 17399 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 14:55:25.590971 17399 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:40611
I20260927 14:55:25.590996 17466 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:40611 every 8 connection(s)
I20260927 14:55:25.591856 17399 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6936m5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790520684602251-392-0/minicluster-data/master-0/data/info.pb
I20260927 14:55:25.592120 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6936m5/build/debug/bin/kudu as pid 17399
I20260927 14:55:25.592247 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6936m5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790520684602251-392-0/minicluster-data/master-0/wal/instance
I20260927 14:55:25.592672 392 external_mini_cluster.cc:1428] Setting key 4a6e3d17c24de9156f1584223746fe4b
I20260927 14:55:25.595191 17467 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 14:55:25.600533 17467 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 544504935a654d3d98f8d69af15791b9: Bootstrap starting.
Sep 27 14:55:25 dist-test-slave-xk1g krb5kdc[17386](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790520925, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 14:55:25.602962 17467 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 544504935a654d3d98f8d69af15791b9: Neither blocks nor log segments found. Creating new log.
I20260927 14:55:25.603646 17467 log.cc:824] T 00000000000000000000000000000000 P 544504935a654d3d98f8d69af15791b9: Log is configured to *not* fsync() on all Append() calls
I20260927 14:55:25.605490 17467 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 544504935a654d3d98f8d69af15791b9: No bootstrap required, opened a new log
I20260927 14:55:25.606295 17470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:55:25.594149 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45060 (local address 127.0.98.62:40611)
0927 14:55:25.594550 (+ 401us) server_negotiation.cc:206] Beginning negotiation
0927 14:55:25.594563 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 14:55:25.594597 (+ 34us) server_negotiation.cc:407] Connection header received
0927 14:55:25.595264 (+ 667us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:55:25.595301 (+ 37us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:55:25.595648 (+ 347us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:55:25.595964 (+ 316us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:55:25.596747 (+ 783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:55:25.597780 (+ 1033us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:55:25.598620 (+ 840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:55:25.599073 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:55:25.599250 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:55:25.601869 (+ 2619us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:55:25.601874 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:55:25.601886 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 14:55:25.601911 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:55:25.603520 (+ 1609us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:55:25.604026 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:55:25.604031 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:55:25.604035 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:55:25.604138 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:55:25.604457 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:55:25.604461 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:55:25.604463 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:55:25.604829 (+ 366us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:55:25.605035 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0927 14:55:25.605322 (+ 287us) server_negotiation.cc:299] Negotiation successful
0927 14:55:25.605627 (+ 305us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":275,"thread_start_us":126,"threads_started":1}
I20260927 14:55:25.609313 17467 raft_consensus.cc:348] T 00000000000000000000000000000000 P 544504935a654d3d98f8d69af15791b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "544504935a654d3d98f8d69af15791b9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40611 } }
I20260927 14:55:25.609509 17467 raft_consensus.cc:374] T 00000000000000000000000000000000 P 544504935a654d3d98f8d69af15791b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 14:55:25.609583 17467 raft_consensus.cc:729] T 00000000000000000000000000000000 P 544504935a654d3d98f8d69af15791b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 544504935a654d3d98f8d69af15791b9, State: Initialized, Role: FOLLOWER
I20260927 14:55:25.609954 17467 consensus_queue.cc:260] T 00000000000000000000000000000000 P 544504935a654d3d98f8d69af15791b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "544504935a654d3d98f8d69af15791b9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40611 } }
I20260927 14:55:25.610097 17467 raft_consensus.cc:388] T 00000000000000000000000000000000 P 544504935a654d3d98f8d69af15791b9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 14:55:25.610158 17467 raft_consensus.cc:482] T 00000000000000000000000000000000 P 544504935a654d3d98f8d69af15791b9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 14:55:25.610275 17467 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 544504935a654d3d98f8d69af15791b9 [term 0 FOLLOWER]: Advancing to term 1
I20260927 14:55:25.611095 17467 raft_consensus.cc:504] T 00000000000000000000000000000000 P 544504935a654d3d98f8d69af15791b9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "544504935a654d3d98f8d69af15791b9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40611 } }
I20260927 14:55:25.611407 17467 leader_election.cc:302] T 00000000000000000000000000000000 P 544504935a654d3d98f8d69af15791b9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 544504935a654d3d98f8d69af15791b9; no voters:
I20260927 14:55:25.611694 17467 leader_election.cc:288] T 00000000000000000000000000000000 P 544504935a654d3d98f8d69af15791b9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 14:55:25.611828 17472 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 544504935a654d3d98f8d69af15791b9 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 14:55:25.612082 17472 raft_consensus.cc:686] T 00000000000000000000000000000000 P 544504935a654d3d98f8d69af15791b9 [term 1 LEADER]: Becoming Leader. State: Replica: 544504935a654d3d98f8d69af15791b9, State: Running, Role: LEADER
I20260927 14:55:25.612399 17472 consensus_queue.cc:237] T 00000000000000000000000000000000 P 544504935a654d3d98f8d69af15791b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "544504935a654d3d98f8d69af15791b9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40611 } }
I20260927 14:55:25.612871 17467 sys_catalog.cc:564] T 00000000000000000000000000000000 P 544504935a654d3d98f8d69af15791b9 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 14:55:25.614080 17474 sys_catalog.cc:455] T 00000000000000000000000000000000 P 544504935a654d3d98f8d69af15791b9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 544504935a654d3d98f8d69af15791b9. Latest consensus state: current_term: 1 leader_uuid: "544504935a654d3d98f8d69af15791b9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "544504935a654d3d98f8d69af15791b9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40611 } } }
I20260927 14:55:25.614090 17473 sys_catalog.cc:455] T 00000000000000000000000000000000 P 544504935a654d3d98f8d69af15791b9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "544504935a654d3d98f8d69af15791b9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "544504935a654d3d98f8d69af15791b9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40611 } } }
I20260927 14:55:25.614219 17474 sys_catalog.cc:458] T 00000000000000000000000000000000 P 544504935a654d3d98f8d69af15791b9 [sys.catalog]: This master's current role is: LEADER
I20260927 14:55:25.614221 17473 sys_catalog.cc:458] T 00000000000000000000000000000000 P 544504935a654d3d98f8d69af15791b9 [sys.catalog]: This master's current role is: LEADER
I20260927 14:55:25.614646 17481 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 14:55:25.617525 17481 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 14:55:25.623219 17481 catalog_manager.cc:1347] Generated new cluster ID: 380a385964174687bbaf2819326d77ce
I20260927 14:55:25.623291 17481 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 14:55:25.640444 17481 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 14:55:25.641422 17481 catalog_manager.cc:1504] Loading token signing keys...
I20260927 14:55:25.646524 17481 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 544504935a654d3d98f8d69af15791b9: Generated new TSK 0
I20260927 14:55:25.647240 17481 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6936m5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790520684602251-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6936m5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790520684602251-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260927 14:55:25.680727 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6936m5/build/debug/bin/kudu
/tmp/dist-test-task6936m5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790520684602251-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790520684602251-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790520684602251-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790520684602251-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40611
--keytab_file=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790520684602251-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41785
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790520684602251-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790520684602251-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790520684602251-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 14:55:25.789892 17493 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 14:55:25.790145 17493 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 14:55:25.790187 17493 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 14:55:25.794173 17493 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 14:55:25.794225 17493 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 14:55:25.794351 17493 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 14:55:25.799470 17493 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41785
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790520684602251-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790520684602251-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790520684602251-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790520684602251-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40611
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17493
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790520684602251-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:15:17 UTC on ccycloud.quasar-ledxfe.root.comops.site
build id 4676134
I20260927 14:55:25.800531 17493 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 14:55:25.801573 17493 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 14:55:25.810879 17493 server_base.cc:1034] running on GCE node
W20260927 14:55:25.810881 17498 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 14:55:25.810839 17499 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 14:55:25.810839 17501 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 14:55:25.811492 17493 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 14:55:25.812100 17493 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 14:55:25.813308 17493 hybrid_clock.cc:648] HybridClock initialized: now 1790520925813272 us; error 45 us; skew 500 ppm
Sep 27 14:55:25 dist-test-slave-xk1g krb5kdc[17386](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790520925, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 14:55:25.816119 17493 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 14:55:25.816962 17493 webserver.cc:460] Webserver started at http://127.0.98.1:39581/ using document root <none> and password file <none>
I20260927 14:55:25.817539 17493 fs_manager.cc:359] Metadata directory not provided
I20260927 14:55:25.817601 17493 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 14:55:25.817772 17493 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 14:55:25.819685 17493 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6936m5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790520684602251-392-0/minicluster-data/ts-0/data/instance:
uuid: "56b62b8f8d5f421993dbde068ddbb967"
format_stamp: "Formatted at 2026-09-27 14:55:25 on dist-test-slave-xk1g"
server_key: "54bfd16c1fa9e2a1e47007b966fc1782"
server_key_iv: "408db684fca3530bd5c7973d987b57ce"
server_key_version: "encryptionkey@0"
I20260927 14:55:25.820188 17493 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6936m5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790520684602251-392-0/minicluster-data/ts-0/wal/instance:
uuid: "56b62b8f8d5f421993dbde068ddbb967"
format_stamp: "Formatted at 2026-09-27 14:55:25 on dist-test-slave-xk1g"
server_key: "54bfd16c1fa9e2a1e47007b966fc1782"
server_key_iv: "408db684fca3530bd5c7973d987b57ce"
server_key_version: "encryptionkey@0"
I20260927 14:55:25.823637 17493 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260927 14:55:25.825887 17508 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 14:55:25.827106 17493 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 14:55:25.827243 17493 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790520684602251-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task6936m5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790520684602251-392-0/minicluster-data/ts-0/wal
uuid: "56b62b8f8d5f421993dbde068ddbb967"
format_stamp: "Formatted at 2026-09-27 14:55:25 on dist-test-slave-xk1g"
server_key: "54bfd16c1fa9e2a1e47007b966fc1782"
server_key_iv: "408db684fca3530bd5c7973d987b57ce"
server_key_version: "encryptionkey@0"
I20260927 14:55:25.827354 17493 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790520684602251-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790520684602251-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790520684602251-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 14:55:25.847628 17493 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 14:55:25.850431 17493 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 14:55:25.850646 17493 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 14:55:25.851203 17493 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 14:55:25.851984 17493 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 14:55:25.852061 17493 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 14:55:25.852129 17493 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 14:55:25.852165 17493 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 14:55:25.860863 17493 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:46549
I20260927 14:55:25.860885 17621 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:46549 every 8 connection(s)
I20260927 14:55:25.861907 17493 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6936m5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790520684602251-392-0/minicluster-data/ts-0/data/info.pb
I20260927 14:55:25.867250 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6936m5/build/debug/bin/kudu as pid 17493
I20260927 14:55:25.867379 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6936m5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790520684602251-392-0/minicluster-data/ts-0/wal/instance
I20260927 14:55:25.867678 392 external_mini_cluster.cc:1428] Setting key 7e95fb463583c88bce5a2d934cd63da8
Sep 27 14:55:25 dist-test-slave-xk1g krb5kdc[17386](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790520925, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 14:55:25.876555 17470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:55:25.863742 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:50513 (local address 127.0.98.62:40611)
0927 14:55:25.863868 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0927 14:55:25.863873 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 14:55:25.864617 (+ 744us) server_negotiation.cc:407] Connection header received
0927 14:55:25.865599 (+ 982us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:55:25.865603 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:55:25.865660 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:55:25.865761 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:55:25.866859 (+ 1098us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:55:25.867739 (+ 880us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:55:25.869378 (+ 1639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:55:25.869812 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:55:25.869933 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:55:25.872460 (+ 2527us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:55:25.872485 (+ 25us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:55:25.872491 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 14:55:25.872533 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:55:25.874306 (+ 1773us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:55:25.874840 (+ 534us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:55:25.874846 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:55:25.874850 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:55:25.874951 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:55:25.875284 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:55:25.875289 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:55:25.875293 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:55:25.875557 (+ 264us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:55:25.875662 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0927 14:55:25.876187 (+ 525us) server_negotiation.cc:299] Negotiation successful
0927 14:55:25.876339 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260927 14:55:25.877342 17624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:55:25.864020 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40611 (local address 127.0.98.1:50513)
0927 14:55:25.864467 (+ 447us) negotiation.cc:107] Waiting for socket to connect
0927 14:55:25.864510 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0927 14:55:25.865378 (+ 868us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:55:25.865925 (+ 547us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:55:25.865932 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:55:25.866269 (+ 337us) client_negotiation.cc:189] Negotiated authn=SASL
0927 14:55:25.866717 (+ 448us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:55:25.866730 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:55:25.867888 (+ 1158us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:55:25.867892 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:55:25.869269 (+ 1377us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:55:25.869279 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:55:25.869952 (+ 673us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:55:25.869955 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:55:25.870084 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:55:25.870720 (+ 636us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 14:55:25.870736 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 14:55:25.872294 (+ 1558us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 14:55:25.874440 (+ 2146us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 14:55:25.874446 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 14:55:25.874458 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0927 14:55:25.874734 (+ 276us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 14:55:25.875050 (+ 316us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 14:55:25.875052 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 14:55:25.875053 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 14:55:25.875189 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 14:55:25.875689 (+ 500us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 14:55:25.875695 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 14:55:25.875938 (+ 243us) client_negotiation.cc:769] Sending connection context
0927 14:55:25.876102 (+ 164us) client_negotiation.cc:240] Negotiation successful
0927 14:55:25.876368 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":276,"thread_start_us":120,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260927 14:55:25.878594 17622 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40611
I20260927 14:55:25.878830 17622 heartbeater.cc:464] Registering TS with master...
I20260927 14:55:25.879317 17622 heartbeater.cc:511] Master 127.0.98.62:40611 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260927 14:55:25.880776 17431 ts_manager.cc:194] Registered new tserver with Master: 56b62b8f8d5f421993dbde068ddbb967 (127.0.98.1:46549)
I20260927 14:55:25.882125 17431 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:50513
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6936m5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790520684602251-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6936m5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790520684602251-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260927 14:55:25.897797 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6936m5/build/debug/bin/kudu
/tmp/dist-test-task6936m5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790520684602251-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790520684602251-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790520684602251-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790520684602251-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40611
--keytab_file=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790520684602251-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41785
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790520684602251-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790520684602251-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790520684602251-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 14:55:26.007333 17627 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 14:55:26.007637 17627 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 14:55:26.007685 17627 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 14:55:26.011801 17627 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 14:55:26.011864 17627 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 14:55:26.011971 17627 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 14:55:26.017601 17627 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41785
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790520684602251-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790520684602251-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790520684602251-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790520684602251-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40611
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17627
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790520684602251-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:15:17 UTC on ccycloud.quasar-ledxfe.root.comops.site
build id 4676134
I20260927 14:55:26.018625 17627 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 14:55:26.019637 17627 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 14:55:26.029044 17635 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 14:55:26.029073 17633 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 14:55:26.029125 17627 server_base.cc:1034] running on GCE node
W20260927 14:55:26.029040 17632 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 14:55:26.029703 17627 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 14:55:26.030362 17627 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 14:55:26.031544 17627 hybrid_clock.cc:648] HybridClock initialized: now 1790520926031504 us; error 54 us; skew 500 ppm
Sep 27 14:55:26 dist-test-slave-xk1g krb5kdc[17386](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790520926, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 14:55:26.034363 17627 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 14:55:26.035262 17627 webserver.cc:460] Webserver started at http://127.0.98.2:41595/ using document root <none> and password file <none>
I20260927 14:55:26.035820 17627 fs_manager.cc:359] Metadata directory not provided
I20260927 14:55:26.035907 17627 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 14:55:26.036140 17627 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 14:55:26.038187 17627 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6936m5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790520684602251-392-0/minicluster-data/ts-1/data/instance:
uuid: "14cc4a4504e34032bfbb8f550bed630f"
format_stamp: "Formatted at 2026-09-27 14:55:26 on dist-test-slave-xk1g"
server_key: "2ad185ff0e33ce13979202d670945b3a"
server_key_iv: "135bcb2b9d1cac6a11c05d3a4cabb703"
server_key_version: "encryptionkey@0"
I20260927 14:55:26.038740 17627 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6936m5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790520684602251-392-0/minicluster-data/ts-1/wal/instance:
uuid: "14cc4a4504e34032bfbb8f550bed630f"
format_stamp: "Formatted at 2026-09-27 14:55:26 on dist-test-slave-xk1g"
server_key: "2ad185ff0e33ce13979202d670945b3a"
server_key_iv: "135bcb2b9d1cac6a11c05d3a4cabb703"
server_key_version: "encryptionkey@0"
I20260927 14:55:26.042384 17627 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.001s
I20260927 14:55:26.044725 17642 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 14:55:26.045977 17627 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260927 14:55:26.046125 17627 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790520684602251-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task6936m5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790520684602251-392-0/minicluster-data/ts-1/wal
uuid: "14cc4a4504e34032bfbb8f550bed630f"
format_stamp: "Formatted at 2026-09-27 14:55:26 on dist-test-slave-xk1g"
server_key: "2ad185ff0e33ce13979202d670945b3a"
server_key_iv: "135bcb2b9d1cac6a11c05d3a4cabb703"
server_key_version: "encryptionkey@0"
I20260927 14:55:26.046240 17627 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790520684602251-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790520684602251-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790520684602251-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 14:55:26.060508 17627 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 14:55:26.063452 17627 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 14:55:26.063656 17627 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 14:55:26.064236 17627 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 14:55:26.065016 17627 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 14:55:26.065117 17627 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 14:55:26.065202 17627 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 14:55:26.065251 17627 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 14:55:26.074051 17627 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:36847
I20260927 14:55:26.074084 17755 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:36847 every 8 connection(s)
I20260927 14:55:26.074890 17627 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6936m5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790520684602251-392-0/minicluster-data/ts-1/data/info.pb
Sep 27 14:55:26 dist-test-slave-xk1g krb5kdc[17386](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790520926, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 14:55:26.083702 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6936m5/build/debug/bin/kudu as pid 17627
I20260927 14:55:26.083832 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6936m5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790520684602251-392-0/minicluster-data/ts-1/wal/instance
I20260927 14:55:26.084189 392 external_mini_cluster.cc:1428] Setting key 00fbafd52419e439bdb828fc5abe7110
I20260927 14:55:26.087445 17470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:55:26.076551 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:55455 (local address 127.0.98.62:40611)
0927 14:55:26.076707 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0927 14:55:26.076711 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 14:55:26.077579 (+ 868us) server_negotiation.cc:407] Connection header received
0927 14:55:26.078387 (+ 808us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:55:26.078390 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:55:26.078434 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:55:26.078507 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:55:26.079568 (+ 1061us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:55:26.080066 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:55:26.081164 (+ 1098us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:55:26.081372 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:55:26.081449 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:55:26.083884 (+ 2435us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:55:26.083888 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:55:26.083890 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 14:55:26.083917 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:55:26.085520 (+ 1603us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:55:26.086033 (+ 513us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:55:26.086037 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:55:26.086039 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:55:26.086120 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:55:26.086447 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:55:26.086452 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:55:26.086454 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:55:26.086626 (+ 172us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:55:26.086738 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0927 14:55:26.087180 (+ 442us) server_negotiation.cc:299] Negotiation successful
0927 14:55:26.087294 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260927 14:55:26.088220 17758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:55:26.076946 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40611 (local address 127.0.98.2:55455)
0927 14:55:26.077434 (+ 488us) negotiation.cc:107] Waiting for socket to connect
0927 14:55:26.077474 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0927 14:55:26.078191 (+ 717us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:55:26.078659 (+ 468us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:55:26.078665 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:55:26.078991 (+ 326us) client_negotiation.cc:189] Negotiated authn=SASL
0927 14:55:26.079450 (+ 459us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:55:26.079463 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:55:26.080166 (+ 703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:55:26.080168 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:55:26.081011 (+ 843us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:55:26.081019 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:55:26.081496 (+ 477us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:55:26.081499 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:55:26.081689 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:55:26.082312 (+ 623us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 14:55:26.082328 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 14:55:26.083684 (+ 1356us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 14:55:26.085628 (+ 1944us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 14:55:26.085634 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 14:55:26.085649 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0927 14:55:26.085926 (+ 277us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 14:55:26.086219 (+ 293us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 14:55:26.086224 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 14:55:26.086230 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0927 14:55:26.086352 (+ 122us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 14:55:26.086726 (+ 374us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 14:55:26.086732 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 14:55:26.086958 (+ 226us) client_negotiation.cc:769] Sending connection context
0927 14:55:26.087140 (+ 182us) client_negotiation.cc:240] Negotiation successful
0927 14:55:26.087447 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":314,"thread_start_us":167,"threads_started":1}
I20260927 14:55:26.089425 17756 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40611
I20260927 14:55:26.089655 17756 heartbeater.cc:464] Registering TS with master...
I20260927 14:55:26.090116 17756 heartbeater.cc:511] Master 127.0.98.62:40611 requested a full tablet report, sending...
I20260927 14:55:26.091197 17431 ts_manager.cc:194] Registered new tserver with Master: 14cc4a4504e34032bfbb8f550bed630f (127.0.98.2:36847)
I20260927 14:55:26.091756 17431 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:55455
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6936m5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790520684602251-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6936m5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790520684602251-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260927 14:55:26.111931 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6936m5/build/debug/bin/kudu
/tmp/dist-test-task6936m5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790520684602251-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790520684602251-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790520684602251-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790520684602251-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40611
--keytab_file=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790520684602251-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41785
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790520684602251-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790520684602251-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790520684602251-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 14:55:26.219728 17761 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 14:55:26.219980 17761 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 14:55:26.220022 17761 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 14:55:26.223960 17761 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 14:55:26.224023 17761 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 14:55:26.224079 17761 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 14:55:26.229249 17761 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41785
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790520684602251-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790520684602251-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790520684602251-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790520684602251-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40611
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17761
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790520684602251-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:15:17 UTC on ccycloud.quasar-ledxfe.root.comops.site
build id 4676134
I20260927 14:55:26.230212 17761 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 14:55:26.231207 17761 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 14:55:26.240727 17766 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 14:55:26.240716 17769 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 14:55:26.240716 17767 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 14:55:26.241165 17761 server_base.cc:1034] running on GCE node
I20260927 14:55:26.241608 17761 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 14:55:26.242255 17761 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 14:55:26.243423 17761 hybrid_clock.cc:648] HybridClock initialized: now 1790520926243410 us; error 31 us; skew 500 ppm
Sep 27 14:55:26 dist-test-slave-xk1g krb5kdc[17386](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790520926, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 14:55:26.246387 17761 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 14:55:26.247251 17761 webserver.cc:460] Webserver started at http://127.0.98.3:35877/ using document root <none> and password file <none>
I20260927 14:55:26.247848 17761 fs_manager.cc:359] Metadata directory not provided
I20260927 14:55:26.247910 17761 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 14:55:26.248080 17761 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 14:55:26.250193 17761 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6936m5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790520684602251-392-0/minicluster-data/ts-2/data/instance:
uuid: "189cd8e00cba42fc94290c61e4bf9ee1"
format_stamp: "Formatted at 2026-09-27 14:55:26 on dist-test-slave-xk1g"
server_key: "50170467c6f485753dc38a7704a125e0"
server_key_iv: "f5d9d074655d6936716798b3721c7a86"
server_key_version: "encryptionkey@0"
I20260927 14:55:26.250685 17761 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6936m5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790520684602251-392-0/minicluster-data/ts-2/wal/instance:
uuid: "189cd8e00cba42fc94290c61e4bf9ee1"
format_stamp: "Formatted at 2026-09-27 14:55:26 on dist-test-slave-xk1g"
server_key: "50170467c6f485753dc38a7704a125e0"
server_key_iv: "f5d9d074655d6936716798b3721c7a86"
server_key_version: "encryptionkey@0"
I20260927 14:55:26.254274 17761 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 14:55:26.256481 17776 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 14:55:26.257722 17761 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 14:55:26.257853 17761 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790520684602251-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task6936m5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790520684602251-392-0/minicluster-data/ts-2/wal
uuid: "189cd8e00cba42fc94290c61e4bf9ee1"
format_stamp: "Formatted at 2026-09-27 14:55:26 on dist-test-slave-xk1g"
server_key: "50170467c6f485753dc38a7704a125e0"
server_key_iv: "f5d9d074655d6936716798b3721c7a86"
server_key_version: "encryptionkey@0"
I20260927 14:55:26.257951 17761 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790520684602251-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790520684602251-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790520684602251-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 14:55:26.270506 17761 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 14:55:26.273100 17761 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 14:55:26.273264 17761 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 14:55:26.273757 17761 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 14:55:26.274470 17761 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 14:55:26.274514 17761 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 14:55:26.274552 17761 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 14:55:26.274573 17761 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 14:55:26.283186 17761 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:39891
I20260927 14:55:26.283212 17889 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:39891 every 8 connection(s)
I20260927 14:55:26.284021 17761 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6936m5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790520684602251-392-0/minicluster-data/ts-2/data/info.pb
I20260927 14:55:26.288393 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6936m5/build/debug/bin/kudu as pid 17761
I20260927 14:55:26.288493 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6936m5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790520684602251-392-0/minicluster-data/ts-2/wal/instance
I20260927 14:55:26.288797 392 external_mini_cluster.cc:1428] Setting key 7a3d2e4decdeaf5f17e9a05d2e8b0fca
Sep 27 14:55:26 dist-test-slave-xk1g krb5kdc[17386](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790520926, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 14:55:26.297155 17470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:55:26.285797 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:39781 (local address 127.0.98.62:40611)
0927 14:55:26.285948 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0927 14:55:26.285951 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 14:55:26.286747 (+ 796us) server_negotiation.cc:407] Connection header received
0927 14:55:26.287650 (+ 903us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:55:26.287653 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:55:26.287711 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:55:26.287789 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:55:26.289033 (+ 1244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:55:26.289712 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:55:26.291011 (+ 1299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:55:26.291244 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:55:26.291353 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:55:26.293758 (+ 2405us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:55:26.293761 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:55:26.293764 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 14:55:26.293790 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:55:26.295146 (+ 1356us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:55:26.295662 (+ 516us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:55:26.295667 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:55:26.295669 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:55:26.295753 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:55:26.296083 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:55:26.296087 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:55:26.296089 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:55:26.296271 (+ 182us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:55:26.296376 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0927 14:55:26.296843 (+ 467us) server_negotiation.cc:299] Negotiation successful
0927 14:55:26.296955 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260927 14:55:26.297894 17892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:55:26.286154 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40611 (local address 127.0.98.3:39781)
0927 14:55:26.286590 (+ 436us) negotiation.cc:107] Waiting for socket to connect
0927 14:55:26.286631 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0927 14:55:26.287422 (+ 791us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:55:26.287955 (+ 533us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:55:26.287961 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:55:26.288291 (+ 330us) client_negotiation.cc:189] Negotiated authn=SASL
0927 14:55:26.288875 (+ 584us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:55:26.288901 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:55:26.289829 (+ 928us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:55:26.289833 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:55:26.290896 (+ 1063us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:55:26.290904 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:55:26.291384 (+ 480us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:55:26.291388 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:55:26.291559 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:55:26.292153 (+ 594us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 14:55:26.292169 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 14:55:26.293604 (+ 1435us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 14:55:26.295295 (+ 1691us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 14:55:26.295300 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 14:55:26.295314 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0927 14:55:26.295559 (+ 245us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 14:55:26.295851 (+ 292us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 14:55:26.295854 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 14:55:26.295856 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 14:55:26.295988 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 14:55:26.296375 (+ 387us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 14:55:26.296381 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 14:55:26.296608 (+ 227us) client_negotiation.cc:769] Sending connection context
0927 14:55:26.296803 (+ 195us) client_negotiation.cc:240] Negotiation successful
0927 14:55:26.297119 (+ 316us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":255,"thread_start_us":106,"threads_started":1}
I20260927 14:55:26.298909 17890 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40611
I20260927 14:55:26.299105 17890 heartbeater.cc:464] Registering TS with master...
I20260927 14:55:26.299504 17890 heartbeater.cc:511] Master 127.0.98.62:40611 requested a full tablet report, sending...
I20260927 14:55:26.300406 17431 ts_manager.cc:194] Registered new tserver with Master: 189cd8e00cba42fc94290c61e4bf9ee1 (127.0.98.3:39891)
I20260927 14:55:26.300911 17431 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:39781
I20260927 14:55:26.302510 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 14:55:26.310976 17470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:55:26.304540 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45072 (local address 127.0.98.62:40611)
0927 14:55:26.304673 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0927 14:55:26.304677 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 14:55:26.304856 (+ 179us) server_negotiation.cc:407] Connection header received
0927 14:55:26.304937 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:55:26.304939 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:55:26.304973 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:55:26.305036 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:55:26.305682 (+ 646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:55:26.306146 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:55:26.306965 (+ 819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:55:26.307164 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:55:26.307255 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:55:26.308376 (+ 1121us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:55:26.308379 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:55:26.308381 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 14:55:26.308400 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:55:26.309602 (+ 1202us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:55:26.309958 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:55:26.309961 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:55:26.309962 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:55:26.310028 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:55:26.310254 (+ 226us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:55:26.310256 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:55:26.310257 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:55:26.310433 (+ 176us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:55:26.310531 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0927 14:55:26.310699 (+ 168us) server_negotiation.cc:299] Negotiation successful
0927 14:55:26.310780 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260927 14:55:26.313653 17431 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45072:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 14:55:26.316574 17431 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 14:55:26.330511 17902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:55:26.324870 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:46549 (local address 127.0.0.1:36694)
0927 14:55:26.325905 (+ 1035us) negotiation.cc:107] Waiting for socket to connect
0927 14:55:26.325958 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0927 14:55:26.326074 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:55:26.326634 (+ 560us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:55:26.326640 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:55:26.326662 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 14:55:26.326813 (+ 151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:55:26.326819 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:55:26.327959 (+ 1140us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:55:26.327964 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:55:26.329200 (+ 1236us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:55:26.329209 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:55:26.329999 (+ 790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:55:26.330002 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:55:26.330084 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:55:26.330112 (+ 28us) client_negotiation.cc:769] Sending connection context
0927 14:55:26.330242 (+ 130us) client_negotiation.cc:240] Negotiation successful
0927 14:55:26.330359 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":965,"thread_start_us":101,"threads_started":1}
I20260927 14:55:26.330526 17904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:55:26.325290 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36694 (local address 127.0.98.1:46549)
0927 14:55:26.325682 (+ 392us) server_negotiation.cc:206] Beginning negotiation
0927 14:55:26.325695 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 14:55:26.326157 (+ 462us) server_negotiation.cc:407] Connection header received
0927 14:55:26.326243 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:55:26.326248 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:55:26.326474 (+ 226us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:55:26.326628 (+ 154us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 14:55:26.326944 (+ 316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:55:26.327834 (+ 890us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:55:26.329315 (+ 1481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:55:26.329889 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:55:26.330001 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:55:26.330143 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0927 14:55:26.330278 (+ 135us) server_negotiation.cc:299] Negotiation successful
0927 14:55:26.330389 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":287,"thread_start_us":90,"threads_started":1}
I20260927 14:55:26.331379 17905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:55:26.324677 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:39891 (local address 127.0.0.1:42988)
0927 14:55:26.325903 (+ 1226us) negotiation.cc:107] Waiting for socket to connect
0927 14:55:26.325955 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0927 14:55:26.326075 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:55:26.327048 (+ 973us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:55:26.327050 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:55:26.327067 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 14:55:26.327148 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:55:26.327154 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:55:26.328480 (+ 1326us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:55:26.328483 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:55:26.329775 (+ 1292us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:55:26.329786 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:55:26.330978 (+ 1192us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:55:26.330982 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:55:26.331067 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:55:26.331089 (+ 22us) client_negotiation.cc:769] Sending connection context
0927 14:55:26.331171 (+ 82us) client_negotiation.cc:240] Negotiation successful
0927 14:55:26.331240 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1049,"mutex_wait_us":24,"thread_start_us":81,"threads_started":1}
I20260927 14:55:26.331782 17901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:55:26.325330 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:36847 (local address 127.0.0.1:42496)
0927 14:55:26.326304 (+ 974us) negotiation.cc:107] Waiting for socket to connect
0927 14:55:26.326318 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 14:55:26.326398 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:55:26.326833 (+ 435us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:55:26.326836 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:55:26.326853 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 14:55:26.326958 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:55:26.326968 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:55:26.328450 (+ 1482us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:55:26.328455 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:55:26.329780 (+ 1325us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:55:26.329786 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:55:26.331388 (+ 1602us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:55:26.331394 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:55:26.331491 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:55:26.331523 (+ 32us) client_negotiation.cc:769] Sending connection context
0927 14:55:26.331590 (+ 67us) client_negotiation.cc:240] Negotiation successful
0927 14:55:26.331667 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":919,"mutex_wait_us":111,"thread_start_us":115,"threads_started":1}
I20260927 14:55:26.331871 17903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:55:26.324833 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42988 (local address 127.0.98.3:39891)
0927 14:55:26.326636 (+ 1803us) server_negotiation.cc:206] Beginning negotiation
0927 14:55:26.326647 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 14:55:26.326667 (+ 20us) server_negotiation.cc:407] Connection header received
0927 14:55:26.326740 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:55:26.326744 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:55:26.326906 (+ 162us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:55:26.327062 (+ 156us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 14:55:26.327252 (+ 190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:55:26.328341 (+ 1089us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:55:26.329905 (+ 1564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:55:26.330691 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:55:26.330792 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:55:26.330917 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0927 14:55:26.331638 (+ 721us) server_negotiation.cc:299] Negotiation successful
0927 14:55:26.331729 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1709,"thread_start_us":140,"threads_started":1}
I20260927 14:55:26.332193 17906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:55:26.325848 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42496 (local address 127.0.98.2:36847)
0927 14:55:26.326299 (+ 451us) server_negotiation.cc:206] Beginning negotiation
0927 14:55:26.326308 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 14:55:26.326393 (+ 85us) server_negotiation.cc:407] Connection header received
0927 14:55:26.326485 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:55:26.326489 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:55:26.326661 (+ 172us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:55:26.326801 (+ 140us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 14:55:26.327091 (+ 290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:55:26.328292 (+ 1201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:55:26.330214 (+ 1922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:55:26.331072 (+ 858us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:55:26.331182 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:55:26.331306 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0927 14:55:26.331934 (+ 628us) server_negotiation.cc:299] Negotiation successful
0927 14:55:26.332030 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":366,"thread_start_us":132,"threads_started":1}
I20260927 14:55:26.333959 17556 tablet_service.cc:1467] Processing CreateTablet for tablet 0dc2a305468c4c5183619fa85f82b490 (DEFAULT_TABLE table=test-table [id=f46846fe1f1b4c7d860a6a7751e8f48d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 14:55:26.334421 17690 tablet_service.cc:1467] Processing CreateTablet for tablet 0dc2a305468c4c5183619fa85f82b490 (DEFAULT_TABLE table=test-table [id=f46846fe1f1b4c7d860a6a7751e8f48d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 14:55:26.335098 17556 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0dc2a305468c4c5183619fa85f82b490. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 14:55:26.334836 17824 tablet_service.cc:1467] Processing CreateTablet for tablet 0dc2a305468c4c5183619fa85f82b490 (DEFAULT_TABLE table=test-table [id=f46846fe1f1b4c7d860a6a7751e8f48d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 14:55:26.335397 17690 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0dc2a305468c4c5183619fa85f82b490. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 14:55:26.335737 17824 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0dc2a305468c4c5183619fa85f82b490. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 14:55:26.342079 17907 tablet_bootstrap.cc:492] T 0dc2a305468c4c5183619fa85f82b490 P 14cc4a4504e34032bfbb8f550bed630f: Bootstrap starting.
I20260927 14:55:26.342729 17908 tablet_bootstrap.cc:492] T 0dc2a305468c4c5183619fa85f82b490 P 56b62b8f8d5f421993dbde068ddbb967: Bootstrap starting.
I20260927 14:55:26.345039 17908 tablet_bootstrap.cc:654] T 0dc2a305468c4c5183619fa85f82b490 P 56b62b8f8d5f421993dbde068ddbb967: Neither blocks nor log segments found. Creating new log.
I20260927 14:55:26.345041 17907 tablet_bootstrap.cc:654] T 0dc2a305468c4c5183619fa85f82b490 P 14cc4a4504e34032bfbb8f550bed630f: Neither blocks nor log segments found. Creating new log.
I20260927 14:55:26.345393 17909 tablet_bootstrap.cc:492] T 0dc2a305468c4c5183619fa85f82b490 P 189cd8e00cba42fc94290c61e4bf9ee1: Bootstrap starting.
I20260927 14:55:26.346362 17908 log.cc:824] T 0dc2a305468c4c5183619fa85f82b490 P 56b62b8f8d5f421993dbde068ddbb967: Log is configured to *not* fsync() on all Append() calls
I20260927 14:55:26.346362 17907 log.cc:824] T 0dc2a305468c4c5183619fa85f82b490 P 14cc4a4504e34032bfbb8f550bed630f: Log is configured to *not* fsync() on all Append() calls
I20260927 14:55:26.347538 17909 tablet_bootstrap.cc:654] T 0dc2a305468c4c5183619fa85f82b490 P 189cd8e00cba42fc94290c61e4bf9ee1: Neither blocks nor log segments found. Creating new log.
I20260927 14:55:26.348192 17907 tablet_bootstrap.cc:492] T 0dc2a305468c4c5183619fa85f82b490 P 14cc4a4504e34032bfbb8f550bed630f: No bootstrap required, opened a new log
I20260927 14:55:26.348377 17908 tablet_bootstrap.cc:492] T 0dc2a305468c4c5183619fa85f82b490 P 56b62b8f8d5f421993dbde068ddbb967: No bootstrap required, opened a new log
I20260927 14:55:26.348389 17907 ts_tablet_manager.cc:1397] T 0dc2a305468c4c5183619fa85f82b490 P 14cc4a4504e34032bfbb8f550bed630f: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.004s
I20260927 14:55:26.348562 17908 ts_tablet_manager.cc:1397] T 0dc2a305468c4c5183619fa85f82b490 P 56b62b8f8d5f421993dbde068ddbb967: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.004s
I20260927 14:55:26.348694 17909 log.cc:824] T 0dc2a305468c4c5183619fa85f82b490 P 189cd8e00cba42fc94290c61e4bf9ee1: Log is configured to *not* fsync() on all Append() calls
I20260927 14:55:26.350651 17909 tablet_bootstrap.cc:492] T 0dc2a305468c4c5183619fa85f82b490 P 189cd8e00cba42fc94290c61e4bf9ee1: No bootstrap required, opened a new log
I20260927 14:55:26.350834 17909 ts_tablet_manager.cc:1397] T 0dc2a305468c4c5183619fa85f82b490 P 189cd8e00cba42fc94290c61e4bf9ee1: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260927 14:55:26.351933 17907 raft_consensus.cc:348] T 0dc2a305468c4c5183619fa85f82b490 P 14cc4a4504e34032bfbb8f550bed630f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "189cd8e00cba42fc94290c61e4bf9ee1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39891 } } peers { permanent_uuid: "56b62b8f8d5f421993dbde068ddbb967" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46549 } } peers { permanent_uuid: "14cc4a4504e34032bfbb8f550bed630f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36847 } }
I20260927 14:55:26.352176 17907 raft_consensus.cc:374] T 0dc2a305468c4c5183619fa85f82b490 P 14cc4a4504e34032bfbb8f550bed630f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 14:55:26.352227 17907 raft_consensus.cc:729] T 0dc2a305468c4c5183619fa85f82b490 P 14cc4a4504e34032bfbb8f550bed630f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 14cc4a4504e34032bfbb8f550bed630f, State: Initialized, Role: FOLLOWER
I20260927 14:55:26.352406 17908 raft_consensus.cc:348] T 0dc2a305468c4c5183619fa85f82b490 P 56b62b8f8d5f421993dbde068ddbb967 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "189cd8e00cba42fc94290c61e4bf9ee1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39891 } } peers { permanent_uuid: "56b62b8f8d5f421993dbde068ddbb967" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46549 } } peers { permanent_uuid: "14cc4a4504e34032bfbb8f550bed630f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36847 } }
I20260927 14:55:26.352631 17908 raft_consensus.cc:374] T 0dc2a305468c4c5183619fa85f82b490 P 56b62b8f8d5f421993dbde068ddbb967 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 14:55:26.352677 17908 raft_consensus.cc:729] T 0dc2a305468c4c5183619fa85f82b490 P 56b62b8f8d5f421993dbde068ddbb967 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 56b62b8f8d5f421993dbde068ddbb967, State: Initialized, Role: FOLLOWER
I20260927 14:55:26.352679 17907 consensus_queue.cc:260] T 0dc2a305468c4c5183619fa85f82b490 P 14cc4a4504e34032bfbb8f550bed630f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "189cd8e00cba42fc94290c61e4bf9ee1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39891 } } peers { permanent_uuid: "56b62b8f8d5f421993dbde068ddbb967" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46549 } } peers { permanent_uuid: "14cc4a4504e34032bfbb8f550bed630f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36847 } }
I20260927 14:55:26.353145 17908 consensus_queue.cc:260] T 0dc2a305468c4c5183619fa85f82b490 P 56b62b8f8d5f421993dbde068ddbb967 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "189cd8e00cba42fc94290c61e4bf9ee1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39891 } } peers { permanent_uuid: "56b62b8f8d5f421993dbde068ddbb967" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46549 } } peers { permanent_uuid: "14cc4a4504e34032bfbb8f550bed630f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36847 } }
I20260927 14:55:26.353642 17756 heartbeater.cc:503] Master 127.0.98.62:40611 was elected leader, sending a full tablet report...
I20260927 14:55:26.353679 17907 ts_tablet_manager.cc:1428] T 0dc2a305468c4c5183619fa85f82b490 P 14cc4a4504e34032bfbb8f550bed630f: Time spent starting tablet: real 0.005s user 0.005s sys 0.001s
I20260927 14:55:26.354049 17908 ts_tablet_manager.cc:1428] T 0dc2a305468c4c5183619fa85f82b490 P 56b62b8f8d5f421993dbde068ddbb967: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260927 14:55:26.354202 17622 heartbeater.cc:503] Master 127.0.98.62:40611 was elected leader, sending a full tablet report...
I20260927 14:55:26.354465 17909 raft_consensus.cc:348] T 0dc2a305468c4c5183619fa85f82b490 P 189cd8e00cba42fc94290c61e4bf9ee1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "189cd8e00cba42fc94290c61e4bf9ee1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39891 } } peers { permanent_uuid: "56b62b8f8d5f421993dbde068ddbb967" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46549 } } peers { permanent_uuid: "14cc4a4504e34032bfbb8f550bed630f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36847 } }
I20260927 14:55:26.354737 17909 raft_consensus.cc:374] T 0dc2a305468c4c5183619fa85f82b490 P 189cd8e00cba42fc94290c61e4bf9ee1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 14:55:26.354792 17909 raft_consensus.cc:729] T 0dc2a305468c4c5183619fa85f82b490 P 189cd8e00cba42fc94290c61e4bf9ee1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 189cd8e00cba42fc94290c61e4bf9ee1, State: Initialized, Role: FOLLOWER
I20260927 14:55:26.355189 17909 consensus_queue.cc:260] T 0dc2a305468c4c5183619fa85f82b490 P 189cd8e00cba42fc94290c61e4bf9ee1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "189cd8e00cba42fc94290c61e4bf9ee1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39891 } } peers { permanent_uuid: "56b62b8f8d5f421993dbde068ddbb967" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46549 } } peers { permanent_uuid: "14cc4a4504e34032bfbb8f550bed630f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36847 } }
I20260927 14:55:26.355803 17890 heartbeater.cc:503] Master 127.0.98.62:40611 was elected leader, sending a full tablet report...
I20260927 14:55:26.356105 17909 ts_tablet_manager.cc:1428] T 0dc2a305468c4c5183619fa85f82b490 P 189cd8e00cba42fc94290c61e4bf9ee1: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
W20260927 14:55:26.362943 17623 tablet.cc:2369] T 0dc2a305468c4c5183619fa85f82b490 P 56b62b8f8d5f421993dbde068ddbb967: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 14:55:26.531250 17913 raft_consensus.cc:482] T 0dc2a305468c4c5183619fa85f82b490 P 14cc4a4504e34032bfbb8f550bed630f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 14:55:26.531505 17913 raft_consensus.cc:504] T 0dc2a305468c4c5183619fa85f82b490 P 14cc4a4504e34032bfbb8f550bed630f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "189cd8e00cba42fc94290c61e4bf9ee1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39891 } } peers { permanent_uuid: "56b62b8f8d5f421993dbde068ddbb967" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46549 } } peers { permanent_uuid: "14cc4a4504e34032bfbb8f550bed630f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36847 } }
I20260927 14:55:26.532742 17913 leader_election.cc:288] T 0dc2a305468c4c5183619fa85f82b490 P 14cc4a4504e34032bfbb8f550bed630f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 189cd8e00cba42fc94290c61e4bf9ee1 (127.0.98.3:39891), 56b62b8f8d5f421993dbde068ddbb967 (127.0.98.1:46549)
W20260927 14:55:26.534936 17891 tablet.cc:2369] T 0dc2a305468c4c5183619fa85f82b490 P 189cd8e00cba42fc94290c61e4bf9ee1: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 14:55:26.536791 17758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:55:26.532964 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:39891 (local address 127.0.98.2:41307)
0927 14:55:26.533179 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0927 14:55:26.533196 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 14:55:26.533316 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:55:26.533648 (+ 332us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:55:26.533651 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:55:26.533668 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 14:55:26.533803 (+ 135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:55:26.533809 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:55:26.534612 (+ 803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:55:26.534616 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:55:26.535628 (+ 1012us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:55:26.535638 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:55:26.536332 (+ 694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:55:26.536336 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:55:26.536464 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:55:26.536486 (+ 22us) client_negotiation.cc:769] Sending connection context
0927 14:55:26.536567 (+ 81us) client_negotiation.cc:240] Negotiation successful
0927 14:55:26.536626 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118,"mutex_wait_us":57}
I20260927 14:55:26.536813 17903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:55:26.533090 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:41307 (local address 127.0.98.3:39891)
0927 14:55:26.533241 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0927 14:55:26.533248 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 14:55:26.533268 (+ 20us) server_negotiation.cc:407] Connection header received
0927 14:55:26.533458 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:55:26.533463 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:55:26.533529 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:55:26.533606 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 14:55:26.533924 (+ 318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:55:26.534505 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:55:26.535754 (+ 1249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:55:26.536214 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:55:26.536295 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:55:26.536348 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0927 14:55:26.536600 (+ 252us) server_negotiation.cc:299] Negotiation successful
0927 14:55:26.536694 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260927 14:55:26.536859 17916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:55:26.533055 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:46549 (local address 127.0.98.2:53819)
0927 14:55:26.533453 (+ 398us) negotiation.cc:107] Waiting for socket to connect
0927 14:55:26.533467 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 14:55:26.533563 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:55:26.533826 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:55:26.533828 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:55:26.533842 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 14:55:26.533978 (+ 136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:55:26.533984 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:55:26.534680 (+ 696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:55:26.534683 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:55:26.535794 (+ 1111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:55:26.535803 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:55:26.536458 (+ 655us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:55:26.536460 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:55:26.536565 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:55:26.536589 (+ 24us) client_negotiation.cc:769] Sending connection context
0927 14:55:26.536664 (+ 75us) client_negotiation.cc:240] Negotiation successful
0927 14:55:26.536725 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":316,"mutex_wait_us":33,"thread_start_us":107,"threads_started":1}
I20260927 14:55:26.536890 17904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:55:26.533176 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:53819 (local address 127.0.98.1:46549)
0927 14:55:26.533307 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0927 14:55:26.533312 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 14:55:26.533548 (+ 236us) server_negotiation.cc:407] Connection header received
0927 14:55:26.533671 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:55:26.533674 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:55:26.533725 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:55:26.533806 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 14:55:26.534085 (+ 279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:55:26.534546 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:55:26.535915 (+ 1369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:55:26.536335 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:55:26.536438 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:55:26.536499 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0927 14:55:26.536695 (+ 196us) server_negotiation.cc:299] Negotiation successful
0927 14:55:26.536773 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260927 14:55:26.537547 17576 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0dc2a305468c4c5183619fa85f82b490" candidate_uuid: "14cc4a4504e34032bfbb8f550bed630f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "56b62b8f8d5f421993dbde068ddbb967" is_pre_election: true
I20260927 14:55:26.537547 17844 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0dc2a305468c4c5183619fa85f82b490" candidate_uuid: "14cc4a4504e34032bfbb8f550bed630f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "189cd8e00cba42fc94290c61e4bf9ee1" is_pre_election: true
I20260927 14:55:26.537863 17576 raft_consensus.cc:2454] T 0dc2a305468c4c5183619fa85f82b490 P 56b62b8f8d5f421993dbde068ddbb967 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 14cc4a4504e34032bfbb8f550bed630f in term 0.
I20260927 14:55:26.537863 17844 raft_consensus.cc:2454] T 0dc2a305468c4c5183619fa85f82b490 P 189cd8e00cba42fc94290c61e4bf9ee1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 14cc4a4504e34032bfbb8f550bed630f in term 0.
I20260927 14:55:26.538388 17646 leader_election.cc:302] T 0dc2a305468c4c5183619fa85f82b490 P 14cc4a4504e34032bfbb8f550bed630f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 14cc4a4504e34032bfbb8f550bed630f, 189cd8e00cba42fc94290c61e4bf9ee1, 56b62b8f8d5f421993dbde068ddbb967; no voters:
I20260927 14:55:26.538698 17913 raft_consensus.cc:2781] T 0dc2a305468c4c5183619fa85f82b490 P 14cc4a4504e34032bfbb8f550bed630f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 14:55:26.538772 17913 raft_consensus.cc:482] T 0dc2a305468c4c5183619fa85f82b490 P 14cc4a4504e34032bfbb8f550bed630f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 14:55:26.538802 17913 raft_consensus.cc:3037] T 0dc2a305468c4c5183619fa85f82b490 P 14cc4a4504e34032bfbb8f550bed630f [term 0 FOLLOWER]: Advancing to term 1
I20260927 14:55:26.539856 17913 raft_consensus.cc:504] T 0dc2a305468c4c5183619fa85f82b490 P 14cc4a4504e34032bfbb8f550bed630f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "189cd8e00cba42fc94290c61e4bf9ee1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39891 } } peers { permanent_uuid: "56b62b8f8d5f421993dbde068ddbb967" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46549 } } peers { permanent_uuid: "14cc4a4504e34032bfbb8f550bed630f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36847 } }
I20260927 14:55:26.540278 17913 leader_election.cc:288] T 0dc2a305468c4c5183619fa85f82b490 P 14cc4a4504e34032bfbb8f550bed630f [CANDIDATE]: Term 1 election: Requested vote from peers 189cd8e00cba42fc94290c61e4bf9ee1 (127.0.98.3:39891), 56b62b8f8d5f421993dbde068ddbb967 (127.0.98.1:46549)
I20260927 14:55:26.540807 17844 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0dc2a305468c4c5183619fa85f82b490" candidate_uuid: "14cc4a4504e34032bfbb8f550bed630f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "189cd8e00cba42fc94290c61e4bf9ee1"
I20260927 14:55:26.540824 17576 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0dc2a305468c4c5183619fa85f82b490" candidate_uuid: "14cc4a4504e34032bfbb8f550bed630f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "56b62b8f8d5f421993dbde068ddbb967"
I20260927 14:55:26.540962 17576 raft_consensus.cc:3037] T 0dc2a305468c4c5183619fa85f82b490 P 56b62b8f8d5f421993dbde068ddbb967 [term 0 FOLLOWER]: Advancing to term 1
I20260927 14:55:26.540938 17844 raft_consensus.cc:3037] T 0dc2a305468c4c5183619fa85f82b490 P 189cd8e00cba42fc94290c61e4bf9ee1 [term 0 FOLLOWER]: Advancing to term 1
I20260927 14:55:26.542029 17576 raft_consensus.cc:2454] T 0dc2a305468c4c5183619fa85f82b490 P 56b62b8f8d5f421993dbde068ddbb967 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 14cc4a4504e34032bfbb8f550bed630f in term 1.
I20260927 14:55:26.542047 17844 raft_consensus.cc:2454] T 0dc2a305468c4c5183619fa85f82b490 P 189cd8e00cba42fc94290c61e4bf9ee1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 14cc4a4504e34032bfbb8f550bed630f in term 1.
I20260927 14:55:26.542390 17646 leader_election.cc:302] T 0dc2a305468c4c5183619fa85f82b490 P 14cc4a4504e34032bfbb8f550bed630f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 14cc4a4504e34032bfbb8f550bed630f, 56b62b8f8d5f421993dbde068ddbb967; no voters:
I20260927 14:55:26.542585 17913 raft_consensus.cc:2781] T 0dc2a305468c4c5183619fa85f82b490 P 14cc4a4504e34032bfbb8f550bed630f [term 1 FOLLOWER]: Leader election won for term 1
I20260927 14:55:26.542784 17913 raft_consensus.cc:686] T 0dc2a305468c4c5183619fa85f82b490 P 14cc4a4504e34032bfbb8f550bed630f [term 1 LEADER]: Becoming Leader. State: Replica: 14cc4a4504e34032bfbb8f550bed630f, State: Running, Role: LEADER
I20260927 14:55:26.543100 17913 consensus_queue.cc:237] T 0dc2a305468c4c5183619fa85f82b490 P 14cc4a4504e34032bfbb8f550bed630f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "189cd8e00cba42fc94290c61e4bf9ee1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39891 } } peers { permanent_uuid: "56b62b8f8d5f421993dbde068ddbb967" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46549 } } peers { permanent_uuid: "14cc4a4504e34032bfbb8f550bed630f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36847 } }
I20260927 14:55:26.546188 17430 catalog_manager.cc:5495] T 0dc2a305468c4c5183619fa85f82b490 P 14cc4a4504e34032bfbb8f550bed630f reported cstate change: term changed from 0 to 1, leader changed from <none> to 14cc4a4504e34032bfbb8f550bed630f (127.0.98.2). New cstate: current_term: 1 leader_uuid: "14cc4a4504e34032bfbb8f550bed630f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "189cd8e00cba42fc94290c61e4bf9ee1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39891 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "56b62b8f8d5f421993dbde068ddbb967" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46549 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "14cc4a4504e34032bfbb8f550bed630f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36847 } health_report { overall_health: HEALTHY } } }
I20260927 14:55:26.563706 17906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:55:26.560298 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42502 (local address 127.0.98.2:36847)
0927 14:55:26.560424 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0927 14:55:26.560444 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0927 14:55:26.560463 (+ 19us) server_negotiation.cc:407] Connection header received
0927 14:55:26.560593 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:55:26.560595 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:55:26.560655 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:55:26.560743 (+ 88us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 14:55:26.560957 (+ 214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:55:26.561512 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:55:26.562434 (+ 922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:55:26.562648 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:55:26.562721 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:55:26.562975 (+ 254us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 14:55:26.563380 (+ 405us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 14:55:26.563468 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0927 14:55:26.563520 (+ 52us) server_negotiation.cc:299] Negotiation successful
0927 14:55:26.563567 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260927 14:55:26.569631 17844 raft_consensus.cc:1261] T 0dc2a305468c4c5183619fa85f82b490 P 189cd8e00cba42fc94290c61e4bf9ee1 [term 1 FOLLOWER]: Refusing update from remote peer 14cc4a4504e34032bfbb8f550bed630f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 14:55:26.569638 17576 raft_consensus.cc:1261] T 0dc2a305468c4c5183619fa85f82b490 P 56b62b8f8d5f421993dbde068ddbb967 [term 1 FOLLOWER]: Refusing update from remote peer 14cc4a4504e34032bfbb8f550bed630f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 14:55:26.570379 17917 consensus_queue.cc:1035] T 0dc2a305468c4c5183619fa85f82b490 P 14cc4a4504e34032bfbb8f550bed630f [LEADER]: Connected to new peer: Peer: permanent_uuid: "56b62b8f8d5f421993dbde068ddbb967" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46549 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 14:55:26.570534 17913 consensus_queue.cc:1035] T 0dc2a305468c4c5183619fa85f82b490 P 14cc4a4504e34032bfbb8f550bed630f [LEADER]: Connected to new peer: Peer: permanent_uuid: "189cd8e00cba42fc94290c61e4bf9ee1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39891 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260927 14:55:26.575925 17757 tablet.cc:2369] T 0dc2a305468c4c5183619fa85f82b490 P 14cc4a4504e34032bfbb8f550bed630f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 14:55:26.577482 17920 mvcc.cc:205] Tried to move back new op lower bound from 7333973715222102016 to 7333973715122864128. Current Snapshot: MvccSnapshot[applied={T|T < 7333973715222102016}]
I20260927 14:55:26.579839 17922 mvcc.cc:205] Tried to move back new op lower bound from 7333973715222102016 to 7333973715122864128. Current Snapshot: MvccSnapshot[applied={T|T < 7333973715222102016 or (T in {7333973715222102016})}]
I20260927 14:55:26.585615 17429 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45072:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 14:55:26.585781 17429 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45072:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 14:55:26.588091 17429 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 544504935a654d3d98f8d69af15791b9: Sending DeleteTablet for 3 replicas of tablet 0dc2a305468c4c5183619fa85f82b490
I20260927 14:55:26.588809 17824 tablet_service.cc:1514] Processing DeleteTablet for tablet 0dc2a305468c4c5183619fa85f82b490 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 14:55:26 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:42988
I20260927 14:55:26.589076 17690 tablet_service.cc:1514] Processing DeleteTablet for tablet 0dc2a305468c4c5183619fa85f82b490 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 14:55:26 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:42496
I20260927 14:55:26.589412 17930 tablet_replica.cc:330] T 0dc2a305468c4c5183619fa85f82b490 P 189cd8e00cba42fc94290c61e4bf9ee1: stopping tablet replica
I20260927 14:55:26.589373 17556 tablet_service.cc:1514] Processing DeleteTablet for tablet 0dc2a305468c4c5183619fa85f82b490 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 14:55:26 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:36694
I20260927 14:55:26.589738 17930 raft_consensus.cc:2229] T 0dc2a305468c4c5183619fa85f82b490 P 189cd8e00cba42fc94290c61e4bf9ee1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 14:55:26.589895 17932 tablet_replica.cc:330] T 0dc2a305468c4c5183619fa85f82b490 P 56b62b8f8d5f421993dbde068ddbb967: stopping tablet replica
I20260927 14:55:26.590068 17930 raft_consensus.cc:2258] T 0dc2a305468c4c5183619fa85f82b490 P 189cd8e00cba42fc94290c61e4bf9ee1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 14:55:26.590121 17931 tablet_replica.cc:330] T 0dc2a305468c4c5183619fa85f82b490 P 14cc4a4504e34032bfbb8f550bed630f: stopping tablet replica
I20260927 14:55:26.590385 17931 raft_consensus.cc:2229] T 0dc2a305468c4c5183619fa85f82b490 P 14cc4a4504e34032bfbb8f550bed630f [term 1 LEADER]: Raft consensus shutting down.
I20260927 14:55:26.590576 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6936m5/build/debug/bin/kudu with pid 17493
I20260927 14:55:26.591805 17931 raft_consensus.cc:2258] T 0dc2a305468c4c5183619fa85f82b490 P 14cc4a4504e34032bfbb8f550bed630f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 14:55:26.592998 17931 ts_tablet_manager.cc:1905] T 0dc2a305468c4c5183619fa85f82b490 P 14cc4a4504e34032bfbb8f550bed630f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 14:55:26.592998 17930 ts_tablet_manager.cc:1905] T 0dc2a305468c4c5183619fa85f82b490 P 189cd8e00cba42fc94290c61e4bf9ee1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 14:55:26.595681 17931 ts_tablet_manager.cc:1918] T 0dc2a305468c4c5183619fa85f82b490 P 14cc4a4504e34032bfbb8f550bed630f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 14:55:26.595681 17930 ts_tablet_manager.cc:1918] T 0dc2a305468c4c5183619fa85f82b490 P 189cd8e00cba42fc94290c61e4bf9ee1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 14:55:26.595782 17930 log.cc:1196] T 0dc2a305468c4c5183619fa85f82b490 P 189cd8e00cba42fc94290c61e4bf9ee1: Deleting WAL directory at /tmp/dist-test-task6936m5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790520684602251-392-0/minicluster-data/ts-2/wal/wals/0dc2a305468c4c5183619fa85f82b490
I20260927 14:55:26.595782 17931 log.cc:1196] T 0dc2a305468c4c5183619fa85f82b490 P 14cc4a4504e34032bfbb8f550bed630f: Deleting WAL directory at /tmp/dist-test-task6936m5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790520684602251-392-0/minicluster-data/ts-1/wal/wals/0dc2a305468c4c5183619fa85f82b490
I20260927 14:55:26.596103 17931 ts_tablet_manager.cc:1939] T 0dc2a305468c4c5183619fa85f82b490 P 14cc4a4504e34032bfbb8f550bed630f: Deleting consensus metadata
I20260927 14:55:26.596115 17930 ts_tablet_manager.cc:1939] T 0dc2a305468c4c5183619fa85f82b490 P 189cd8e00cba42fc94290c61e4bf9ee1: Deleting consensus metadata
I20260927 14:55:26.597265 17417 catalog_manager.cc:4841] TS 14cc4a4504e34032bfbb8f550bed630f (127.0.98.2:36847): tablet 0dc2a305468c4c5183619fa85f82b490 (table test-table [id=f46846fe1f1b4c7d860a6a7751e8f48d]) successfully deleted
I20260927 14:55:26.597265 17416 catalog_manager.cc:4841] TS 189cd8e00cba42fc94290c61e4bf9ee1 (127.0.98.3:39891): tablet 0dc2a305468c4c5183619fa85f82b490 (table test-table [id=f46846fe1f1b4c7d860a6a7751e8f48d]) successfully deleted
W20260927 14:55:26.602581 17418 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:46549 (error 108)
I20260927 14:55:26.603583 17901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:55:26.603310 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:46549 (local address 127.0.0.1:36702)
0927 14:55:26.603440 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0927 14:55:26.603505 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:46549: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260927 14:55:26.603746 17418 catalog_manager.cc:4584] TS 56b62b8f8d5f421993dbde068ddbb967 (127.0.98.1:46549): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 0dc2a305468c4c5183619fa85f82b490: Network error: Client connection negotiation failed: client connection to 127.0.98.1:46549: connect: Connection refused (error 111)
I20260927 14:55:26.604187 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6936m5/build/debug/bin/kudu with pid 17627
I20260927 14:55:26.610327 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6936m5/build/debug/bin/kudu with pid 17761
I20260927 14:55:26.616274 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6936m5/build/debug/bin/kudu with pid 17399
2026-09-27T14:55:26Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (2355 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-09-27T14:55:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T14:55:26Z Disabled control of system clock
I20260927 14:55:26.645534 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6936m5/build/debug/bin/kudu
/tmp/dist-test-task6936m5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790520684602251-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790520684602251-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790520684602251-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790520684602251-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33731
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:33007
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:33731
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260927 14:55:26.751832 17937 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 14:55:26.752077 17937 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 14:55:26.752112 17937 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 14:55:26.756093 17937 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 14:55:26.756148 17937 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 14:55:26.756165 17937 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 14:55:26.756181 17937 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 14:55:26.756196 17937 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 14:55:26.760792 17937 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33007
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790520684602251-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790520684602251-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:33731
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33731
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790520684602251-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17937
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790520684602251-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:15:17 UTC on ccycloud.quasar-ledxfe.root.comops.site
build id 4676134
I20260927 14:55:26.761857 17937 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 14:55:26.763123 17937 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 14:55:26.771528 17943 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 14:55:26.771505 17945 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 14:55:26.771498 17942 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 14:55:26.771727 17937 server_base.cc:1034] running on GCE node
I20260927 14:55:26.772378 17937 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 14:55:26.773521 17937 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 14:55:26.774761 17937 hybrid_clock.cc:648] HybridClock initialized: now 1790520926774745 us; error 27 us; skew 500 ppm
I20260927 14:55:26.776655 17937 webserver.cc:460] Webserver started at http://127.0.98.62:35293/ using document root <none> and password file <none>
I20260927 14:55:26.777330 17937 fs_manager.cc:359] Metadata directory not provided
I20260927 14:55:26.777424 17937 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 14:55:26.777663 17937 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 14:55:26.779930 17937 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6936m5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790520684602251-392-0/minicluster-data/master-0/data/instance:
uuid: "3cc5b5bd948c4f6ead0c0f1287d40cd1"
format_stamp: "Formatted at 2026-09-27 14:55:26 on dist-test-slave-xk1g"
server_key: "a25f8f4d9264c27f126481a3ab61a994"
server_key_iv: "62fc11404cd7a1492a48f4ef989b2997"
server_key_version: "encryptionkey@0"
I20260927 14:55:26.780560 17937 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6936m5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790520684602251-392-0/minicluster-data/master-0/wal/instance:
uuid: "3cc5b5bd948c4f6ead0c0f1287d40cd1"
format_stamp: "Formatted at 2026-09-27 14:55:26 on dist-test-slave-xk1g"
server_key: "a25f8f4d9264c27f126481a3ab61a994"
server_key_iv: "62fc11404cd7a1492a48f4ef989b2997"
server_key_version: "encryptionkey@0"
I20260927 14:55:26.784487 17937 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260927 14:55:26.787045 17951 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 14:55:26.788273 17937 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 14:55:26.788416 17937 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790520684602251-392-0/minicluster-data/master-0/data,/tmp/dist-test-task6936m5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790520684602251-392-0/minicluster-data/master-0/wal
uuid: "3cc5b5bd948c4f6ead0c0f1287d40cd1"
format_stamp: "Formatted at 2026-09-27 14:55:26 on dist-test-slave-xk1g"
server_key: "a25f8f4d9264c27f126481a3ab61a994"
server_key_iv: "62fc11404cd7a1492a48f4ef989b2997"
server_key_version: "encryptionkey@0"
I20260927 14:55:26.788530 17937 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790520684602251-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790520684602251-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790520684602251-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 14:55:26.797835 17937 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 14:55:26.798448 17937 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 14:55:26.798636 17937 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 14:55:26.805146 17937 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:33731
I20260927 14:55:26.805131 18003 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:33731 every 8 connection(s)
I20260927 14:55:26.806093 17937 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6936m5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790520684602251-392-0/minicluster-data/master-0/data/info.pb
I20260927 14:55:26.809144 18004 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 14:55:26.810648 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6936m5/build/debug/bin/kudu as pid 17937
I20260927 14:55:26.810794 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6936m5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790520684602251-392-0/minicluster-data/master-0/wal/instance
I20260927 14:55:26.811053 392 external_mini_cluster.cc:1428] Setting key 8875a567b84ee855384eab89814b83be
I20260927 14:55:26.815423 18004 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3cc5b5bd948c4f6ead0c0f1287d40cd1: Bootstrap starting.
I20260927 14:55:26.817629 18004 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3cc5b5bd948c4f6ead0c0f1287d40cd1: Neither blocks nor log segments found. Creating new log.
I20260927 14:55:26.818331 18004 log.cc:824] T 00000000000000000000000000000000 P 3cc5b5bd948c4f6ead0c0f1287d40cd1: Log is configured to *not* fsync() on all Append() calls
I20260927 14:55:26.820441 18004 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3cc5b5bd948c4f6ead0c0f1287d40cd1: No bootstrap required, opened a new log
I20260927 14:55:26.820528 18007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:55:26.812504 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55778 (local address 127.0.98.62:33731)
0927 14:55:26.813231 (+ 727us) server_negotiation.cc:206] Beginning negotiation
0927 14:55:26.813247 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0927 14:55:26.813284 (+ 37us) server_negotiation.cc:407] Connection header received
0927 14:55:26.813956 (+ 672us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:55:26.813974 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:55:26.814361 (+ 387us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:55:26.814689 (+ 328us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:55:26.815116 (+ 427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:55:26.816110 (+ 994us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:55:26.817000 (+ 890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:55:26.817529 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:55:26.817703 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:55:26.818560 (+ 857us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:55:26.818566 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:55:26.818582 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 14:55:26.819035 (+ 453us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:55:26.819104 (+ 69us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 14:55:26.819132 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:55:26.819236 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0927 14:55:26.819401 (+ 165us) server_negotiation.cc:299] Negotiation successful
0927 14:55:26.819653 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":318,"thread_start_us":148,"threads_started":1}
I20260927 14:55:26.823786 18004 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3cc5b5bd948c4f6ead0c0f1287d40cd1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3cc5b5bd948c4f6ead0c0f1287d40cd1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33731 } }
I20260927 14:55:26.824010 18004 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3cc5b5bd948c4f6ead0c0f1287d40cd1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 14:55:26.824100 18004 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3cc5b5bd948c4f6ead0c0f1287d40cd1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3cc5b5bd948c4f6ead0c0f1287d40cd1, State: Initialized, Role: FOLLOWER
I20260927 14:55:26.824584 18004 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3cc5b5bd948c4f6ead0c0f1287d40cd1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3cc5b5bd948c4f6ead0c0f1287d40cd1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33731 } }
I20260927 14:55:26.824705 18004 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3cc5b5bd948c4f6ead0c0f1287d40cd1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 14:55:26.824755 18004 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3cc5b5bd948c4f6ead0c0f1287d40cd1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 14:55:26.824822 18004 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3cc5b5bd948c4f6ead0c0f1287d40cd1 [term 0 FOLLOWER]: Advancing to term 1
I20260927 14:55:26.825666 18004 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3cc5b5bd948c4f6ead0c0f1287d40cd1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3cc5b5bd948c4f6ead0c0f1287d40cd1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33731 } }
I20260927 14:55:26.825935 18004 leader_election.cc:302] T 00000000000000000000000000000000 P 3cc5b5bd948c4f6ead0c0f1287d40cd1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3cc5b5bd948c4f6ead0c0f1287d40cd1; no voters:
I20260927 14:55:26.826169 18004 leader_election.cc:288] T 00000000000000000000000000000000 P 3cc5b5bd948c4f6ead0c0f1287d40cd1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 14:55:26.826290 18009 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3cc5b5bd948c4f6ead0c0f1287d40cd1 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 14:55:26.826508 18009 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3cc5b5bd948c4f6ead0c0f1287d40cd1 [term 1 LEADER]: Becoming Leader. State: Replica: 3cc5b5bd948c4f6ead0c0f1287d40cd1, State: Running, Role: LEADER
I20260927 14:55:26.826786 18009 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3cc5b5bd948c4f6ead0c0f1287d40cd1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3cc5b5bd948c4f6ead0c0f1287d40cd1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33731 } }
I20260927 14:55:26.827291 18004 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3cc5b5bd948c4f6ead0c0f1287d40cd1 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 14:55:26.828394 18011 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3cc5b5bd948c4f6ead0c0f1287d40cd1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3cc5b5bd948c4f6ead0c0f1287d40cd1. Latest consensus state: current_term: 1 leader_uuid: "3cc5b5bd948c4f6ead0c0f1287d40cd1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3cc5b5bd948c4f6ead0c0f1287d40cd1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33731 } } }
I20260927 14:55:26.828418 18010 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3cc5b5bd948c4f6ead0c0f1287d40cd1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3cc5b5bd948c4f6ead0c0f1287d40cd1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3cc5b5bd948c4f6ead0c0f1287d40cd1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33731 } } }
I20260927 14:55:26.828523 18011 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3cc5b5bd948c4f6ead0c0f1287d40cd1 [sys.catalog]: This master's current role is: LEADER
I20260927 14:55:26.828522 18010 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3cc5b5bd948c4f6ead0c0f1287d40cd1 [sys.catalog]: This master's current role is: LEADER
I20260927 14:55:26.828869 18018 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 14:55:26.831658 18018 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 14:55:26.836485 18018 catalog_manager.cc:1347] Generated new cluster ID: 6e404b37746444968347b680c012133a
I20260927 14:55:26.836540 18018 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 14:55:26.843258 18018 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 14:55:26.844125 18018 catalog_manager.cc:1504] Loading token signing keys...
I20260927 14:55:26.854761 18018 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3cc5b5bd948c4f6ead0c0f1287d40cd1: Generated new TSK 0
I20260927 14:55:26.855489 18018 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 14:55:26.861519 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6936m5/build/debug/bin/kudu
/tmp/dist-test-task6936m5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790520684602251-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790520684602251-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790520684602251-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790520684602251-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33731
--builtin_ntp_servers=127.0.98.20:33007
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260927 14:55:26.974112 18028 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 14:55:26.974391 18028 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 14:55:26.974475 18028 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 14:55:26.978529 18028 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 14:55:26.978600 18028 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 14:55:26.978686 18028 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 14:55:26.983376 18028 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33007
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790520684602251-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790520684602251-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790520684602251-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33731
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18028
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790520684602251-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:15:17 UTC on ccycloud.quasar-ledxfe.root.comops.site
build id 4676134
I20260927 14:55:26.984434 18028 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 14:55:26.985881 18028 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 14:55:26.995549 18036 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 14:55:26.995528 18033 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 14:55:26.995528 18034 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 14:55:26.995738 18028 server_base.cc:1034] running on GCE node
I20260927 14:55:26.996258 18028 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 14:55:26.996855 18028 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 14:55:26.998039 18028 hybrid_clock.cc:648] HybridClock initialized: now 1790520926998030 us; error 58 us; skew 500 ppm
I20260927 14:55:26.999862 18028 webserver.cc:460] Webserver started at http://127.0.98.1:39269/ using document root <none> and password file <none>
I20260927 14:55:27.000485 18028 fs_manager.cc:359] Metadata directory not provided
I20260927 14:55:27.000581 18028 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 14:55:27.000813 18028 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 14:55:27.002939 18028 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6936m5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790520684602251-392-0/minicluster-data/ts-0/data/instance:
uuid: "ba5c9e0c0800457993170da02d0ad5bc"
format_stamp: "Formatted at 2026-09-27 14:55:26 on dist-test-slave-xk1g"
server_key: "90b1eb4433c92e1418d801767a5fbe56"
server_key_iv: "2e4f0e882f5b5a4289b08d014c46c333"
server_key_version: "encryptionkey@0"
I20260927 14:55:27.003504 18028 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6936m5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790520684602251-392-0/minicluster-data/ts-0/wal/instance:
uuid: "ba5c9e0c0800457993170da02d0ad5bc"
format_stamp: "Formatted at 2026-09-27 14:55:26 on dist-test-slave-xk1g"
server_key: "90b1eb4433c92e1418d801767a5fbe56"
server_key_iv: "2e4f0e882f5b5a4289b08d014c46c333"
server_key_version: "encryptionkey@0"
I20260927 14:55:27.007380 18028 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260927 14:55:27.009914 18042 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 14:55:27.011111 18028 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 14:55:27.011261 18028 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790520684602251-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task6936m5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790520684602251-392-0/minicluster-data/ts-0/wal
uuid: "ba5c9e0c0800457993170da02d0ad5bc"
format_stamp: "Formatted at 2026-09-27 14:55:26 on dist-test-slave-xk1g"
server_key: "90b1eb4433c92e1418d801767a5fbe56"
server_key_iv: "2e4f0e882f5b5a4289b08d014c46c333"
server_key_version: "encryptionkey@0"
I20260927 14:55:27.011386 18028 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790520684602251-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790520684602251-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790520684602251-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 14:55:27.025862 18028 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 14:55:27.026571 18028 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 14:55:27.026774 18028 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 14:55:27.027354 18028 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 14:55:27.028376 18028 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 14:55:27.028448 18028 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 14:55:27.028520 18028 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 14:55:27.028553 18028 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 14:55:27.036875 18028 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:37635
I20260927 14:55:27.036970 18155 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:37635 every 8 connection(s)
I20260927 14:55:27.037797 18028 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6936m5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790520684602251-392-0/minicluster-data/ts-0/data/info.pb
I20260927 14:55:27.046584 18007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:55:27.039676 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:47151 (local address 127.0.98.62:33731)
0927 14:55:27.039850 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0927 14:55:27.039855 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 14:55:27.040877 (+ 1022us) server_negotiation.cc:407] Connection header received
0927 14:55:27.042239 (+ 1362us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:55:27.042243 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:55:27.042289 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:55:27.042366 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:55:27.043221 (+ 855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:55:27.043759 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:55:27.044852 (+ 1093us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:55:27.045124 (+ 272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:55:27.045208 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:55:27.046185 (+ 977us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:55:27.046187 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:55:27.046189 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 14:55:27.046201 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:55:27.046217 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 14:55:27.046224 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:55:27.046280 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0927 14:55:27.046415 (+ 135us) server_negotiation.cc:299] Negotiation successful
0927 14:55:27.046470 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260927 14:55:27.047438 18158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:55:27.040089 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33731 (local address 127.0.98.1:47151)
0927 14:55:27.040720 (+ 631us) negotiation.cc:107] Waiting for socket to connect
0927 14:55:27.040761 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0927 14:55:27.042036 (+ 1275us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:55:27.042532 (+ 496us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:55:27.042538 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:55:27.042619 (+ 81us) client_negotiation.cc:189] Negotiated authn=SASL
0927 14:55:27.043088 (+ 469us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:55:27.043100 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:55:27.043868 (+ 768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:55:27.043870 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:55:27.044751 (+ 881us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:55:27.044760 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:55:27.045253 (+ 493us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:55:27.045256 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:55:27.045389 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:55:27.045945 (+ 556us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 14:55:27.045947 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0927 14:55:27.045985 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 14:55:27.046045 (+ 60us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 14:55:27.046295 (+ 250us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 14:55:27.046302 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 14:55:27.046306 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 14:55:27.046397 (+ 91us) client_negotiation.cc:240] Negotiation successful
0927 14:55:27.046607 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":255,"thread_start_us":117,"threads_started":1}
I20260927 14:55:27.047823 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6936m5/build/debug/bin/kudu as pid 18028
I20260927 14:55:27.047952 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6936m5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790520684602251-392-0/minicluster-data/ts-0/wal/instance
I20260927 14:55:27.048316 392 external_mini_cluster.cc:1428] Setting key ba9bc16e19e3043e32f22b5c5075947c
I20260927 14:55:27.048774 18156 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33731
I20260927 14:55:27.049014 18156 heartbeater.cc:464] Registering TS with master...
I20260927 14:55:27.049530 18156 heartbeater.cc:511] Master 127.0.98.62:33731 requested a full tablet report, sending...
I20260927 14:55:27.050303 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6936m5/build/debug/bin/kudu
/tmp/dist-test-task6936m5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790520684602251-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790520684602251-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790520684602251-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790520684602251-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33731
--builtin_ntp_servers=127.0.98.20:33007
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260927 14:55:27.050976 17968 ts_manager.cc:194] Registered new tserver with Master: ba5c9e0c0800457993170da02d0ad5bc (127.0.98.1:37635)
I20260927 14:55:27.052068 17968 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.1:47151
W20260927 14:55:27.156700 18159 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 14:55:27.156921 18159 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 14:55:27.156955 18159 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 14:55:27.160905 18159 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 14:55:27.160950 18159 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 14:55:27.161001 18159 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 14:55:27.165637 18159 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33007
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790520684602251-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790520684602251-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790520684602251-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33731
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18159
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790520684602251-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:15:17 UTC on ccycloud.quasar-ledxfe.root.comops.site
build id 4676134
I20260927 14:55:27.166558 18159 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 14:55:27.167805 18159 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 14:55:27.177124 18165 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 14:55:27.177186 18159 server_base.cc:1034] running on GCE node
W20260927 14:55:27.177129 18164 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 14:55:27.177191 18167 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 14:55:27.177747 18159 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 14:55:27.178354 18159 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 14:55:27.179534 18159 hybrid_clock.cc:648] HybridClock initialized: now 1790520927179526 us; error 33 us; skew 500 ppm
I20260927 14:55:27.181424 18159 webserver.cc:460] Webserver started at http://127.0.98.2:42255/ using document root <none> and password file <none>
I20260927 14:55:27.181993 18159 fs_manager.cc:359] Metadata directory not provided
I20260927 14:55:27.182085 18159 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 14:55:27.182312 18159 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 14:55:27.184320 18159 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6936m5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790520684602251-392-0/minicluster-data/ts-1/data/instance:
uuid: "e043ff41b9ed40c4934257e6a098a0a9"
format_stamp: "Formatted at 2026-09-27 14:55:27 on dist-test-slave-xk1g"
server_key: "4aee0d82b817118d528815a8a7b2887d"
server_key_iv: "4e8babdf3dbacdfa8ebac1c15f51e768"
server_key_version: "encryptionkey@0"
I20260927 14:55:27.184885 18159 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6936m5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790520684602251-392-0/minicluster-data/ts-1/wal/instance:
uuid: "e043ff41b9ed40c4934257e6a098a0a9"
format_stamp: "Formatted at 2026-09-27 14:55:27 on dist-test-slave-xk1g"
server_key: "4aee0d82b817118d528815a8a7b2887d"
server_key_iv: "4e8babdf3dbacdfa8ebac1c15f51e768"
server_key_version: "encryptionkey@0"
I20260927 14:55:27.188460 18159 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.002s
I20260927 14:55:27.190817 18173 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 14:55:27.192039 18159 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s
I20260927 14:55:27.192160 18159 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790520684602251-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task6936m5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790520684602251-392-0/minicluster-data/ts-1/wal
uuid: "e043ff41b9ed40c4934257e6a098a0a9"
format_stamp: "Formatted at 2026-09-27 14:55:27 on dist-test-slave-xk1g"
server_key: "4aee0d82b817118d528815a8a7b2887d"
server_key_iv: "4e8babdf3dbacdfa8ebac1c15f51e768"
server_key_version: "encryptionkey@0"
I20260927 14:55:27.192269 18159 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790520684602251-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790520684602251-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790520684602251-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 14:55:27.222201 18159 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 14:55:27.222874 18159 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 14:55:27.223069 18159 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 14:55:27.223627 18159 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 14:55:27.224548 18159 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 14:55:27.224619 18159 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 14:55:27.224689 18159 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 14:55:27.224740 18159 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 14:55:27.233855 18159 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:43925
I20260927 14:55:27.233904 18286 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:43925 every 8 connection(s)
I20260927 14:55:27.234699 18159 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6936m5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790520684602251-392-0/minicluster-data/ts-1/data/info.pb
I20260927 14:55:27.236227 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6936m5/build/debug/bin/kudu as pid 18159
I20260927 14:55:27.236344 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6936m5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790520684602251-392-0/minicluster-data/ts-1/wal/instance
I20260927 14:55:27.236652 392 external_mini_cluster.cc:1428] Setting key 60c427a8923d3ba778a23f828d98a257
I20260927 14:55:27.238686 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6936m5/build/debug/bin/kudu
/tmp/dist-test-task6936m5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790520684602251-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790520684602251-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790520684602251-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790520684602251-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33731
--builtin_ntp_servers=127.0.98.20:33007
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260927 14:55:27.244321 18007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:55:27.236513 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:57223 (local address 127.0.98.62:33731)
0927 14:55:27.236663 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0927 14:55:27.236670 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 14:55:27.237903 (+ 1233us) server_negotiation.cc:407] Connection header received
0927 14:55:27.239383 (+ 1480us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:55:27.239389 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:55:27.239449 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:55:27.239546 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:55:27.240665 (+ 1119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:55:27.241239 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:55:27.242394 (+ 1155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:55:27.242619 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:55:27.242736 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:55:27.243824 (+ 1088us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:55:27.243830 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:55:27.243838 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 14:55:27.243849 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:55:27.243867 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 14:55:27.243875 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:55:27.243949 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0927 14:55:27.244107 (+ 158us) server_negotiation.cc:299] Negotiation successful
0927 14:55:27.244170 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260927 14:55:27.245175 18289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:55:27.236961 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33731 (local address 127.0.98.2:57223)
0927 14:55:27.237712 (+ 751us) negotiation.cc:107] Waiting for socket to connect
0927 14:55:27.237763 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0927 14:55:27.239171 (+ 1408us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:55:27.239688 (+ 517us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:55:27.239694 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:55:27.239776 (+ 82us) client_negotiation.cc:189] Negotiated authn=SASL
0927 14:55:27.240493 (+ 717us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:55:27.240514 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:55:27.241364 (+ 850us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:55:27.241367 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:55:27.242267 (+ 900us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:55:27.242277 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:55:27.242767 (+ 490us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:55:27.242771 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:55:27.242942 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:55:27.243581 (+ 639us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 14:55:27.243583 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0927 14:55:27.243620 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 14:55:27.243679 (+ 59us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 14:55:27.243962 (+ 283us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 14:55:27.243972 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 14:55:27.243978 (+ 6us) client_negotiation.cc:769] Sending connection context
0927 14:55:27.244107 (+ 129us) client_negotiation.cc:240] Negotiation successful
0927 14:55:27.244390 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":299,"thread_start_us":125,"threads_started":1}
I20260927 14:55:27.246373 18287 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33731
I20260927 14:55:27.246594 18287 heartbeater.cc:464] Registering TS with master...
I20260927 14:55:27.247068 18287 heartbeater.cc:511] Master 127.0.98.62:33731 requested a full tablet report, sending...
I20260927 14:55:27.248201 17968 ts_manager.cc:194] Registered new tserver with Master: e043ff41b9ed40c4934257e6a098a0a9 (127.0.98.2:43925)
I20260927 14:55:27.249087 17968 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.2:57223
W20260927 14:55:27.350999 18290 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 14:55:27.351275 18290 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 14:55:27.351313 18290 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 14:55:27.355415 18290 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 14:55:27.355465 18290 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 14:55:27.355517 18290 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 14:55:27.360110 18290 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33007
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790520684602251-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790520684602251-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790520684602251-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33731
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18290
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790520684602251-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:15:17 UTC on ccycloud.quasar-ledxfe.root.comops.site
build id 4676134
I20260927 14:55:27.361094 18290 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 14:55:27.362409 18290 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 14:55:27.371536 18295 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 14:55:27.371544 18296 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 14:55:27.371740 18290 server_base.cc:1034] running on GCE node
W20260927 14:55:27.371892 18298 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 14:55:27.372298 18290 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 14:55:27.372965 18290 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 14:55:27.374154 18290 hybrid_clock.cc:648] HybridClock initialized: now 1790520927374117 us; error 47 us; skew 500 ppm
I20260927 14:55:27.375810 18290 webserver.cc:460] Webserver started at http://127.0.98.3:34271/ using document root <none> and password file <none>
I20260927 14:55:27.376324 18290 fs_manager.cc:359] Metadata directory not provided
I20260927 14:55:27.376381 18290 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 14:55:27.376533 18290 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 14:55:27.378526 18290 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6936m5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790520684602251-392-0/minicluster-data/ts-2/data/instance:
uuid: "12b88a1e5a13473a907d6372bc7bc90f"
format_stamp: "Formatted at 2026-09-27 14:55:27 on dist-test-slave-xk1g"
server_key: "738605a6f2694f51ea4323821597810d"
server_key_iv: "134b57b61c1cc229b17108622bb287e9"
server_key_version: "encryptionkey@0"
I20260927 14:55:27.379015 18290 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6936m5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790520684602251-392-0/minicluster-data/ts-2/wal/instance:
uuid: "12b88a1e5a13473a907d6372bc7bc90f"
format_stamp: "Formatted at 2026-09-27 14:55:27 on dist-test-slave-xk1g"
server_key: "738605a6f2694f51ea4323821597810d"
server_key_iv: "134b57b61c1cc229b17108622bb287e9"
server_key_version: "encryptionkey@0"
I20260927 14:55:27.382444 18290 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 14:55:27.384773 18304 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 14:55:27.385972 18290 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 14:55:27.386097 18290 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790520684602251-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task6936m5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790520684602251-392-0/minicluster-data/ts-2/wal
uuid: "12b88a1e5a13473a907d6372bc7bc90f"
format_stamp: "Formatted at 2026-09-27 14:55:27 on dist-test-slave-xk1g"
server_key: "738605a6f2694f51ea4323821597810d"
server_key_iv: "134b57b61c1cc229b17108622bb287e9"
server_key_version: "encryptionkey@0"
I20260927 14:55:27.386200 18290 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790520684602251-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790520684602251-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790520684602251-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 14:55:27.408114 18290 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 14:55:27.408759 18290 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 14:55:27.408943 18290 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 14:55:27.409510 18290 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 14:55:27.410406 18290 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 14:55:27.410457 18290 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 14:55:27.410537 18290 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 14:55:27.410573 18290 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 14:55:27.419188 18290 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:39519
I20260927 14:55:27.419215 18417 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:39519 every 8 connection(s)
I20260927 14:55:27.420043 18290 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6936m5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790520684602251-392-0/minicluster-data/ts-2/data/info.pb
I20260927 14:55:27.425369 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6936m5/build/debug/bin/kudu as pid 18290
I20260927 14:55:27.425455 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6936m5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790520684602251-392-0/minicluster-data/ts-2/wal/instance
I20260927 14:55:27.425719 392 external_mini_cluster.cc:1428] Setting key 59ac2f8cd843657bc06909a83fbdab27
I20260927 14:55:27.429260 18007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:55:27.421798 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:44635 (local address 127.0.98.62:33731)
0927 14:55:27.421929 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0927 14:55:27.421933 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 14:55:27.422873 (+ 940us) server_negotiation.cc:407] Connection header received
0927 14:55:27.424250 (+ 1377us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:55:27.424254 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:55:27.424320 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:55:27.424419 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:55:27.425302 (+ 883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:55:27.425948 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:55:27.427154 (+ 1206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:55:27.427469 (+ 315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:55:27.427592 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:55:27.428817 (+ 1225us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:55:27.428820 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:55:27.428822 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 14:55:27.428833 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:55:27.428849 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 14:55:27.428857 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:55:27.428914 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0927 14:55:27.429105 (+ 191us) server_negotiation.cc:299] Negotiation successful
0927 14:55:27.429152 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260927 14:55:27.430058 18420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:55:27.422104 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33731 (local address 127.0.98.3:44635)
0927 14:55:27.422712 (+ 608us) negotiation.cc:107] Waiting for socket to connect
0927 14:55:27.422759 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0927 14:55:27.424017 (+ 1258us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:55:27.424589 (+ 572us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:55:27.424595 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:55:27.424675 (+ 80us) client_negotiation.cc:189] Negotiated authn=SASL
0927 14:55:27.425147 (+ 472us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:55:27.425174 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:55:27.426075 (+ 901us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:55:27.426078 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:55:27.427037 (+ 959us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:55:27.427047 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:55:27.427629 (+ 582us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:55:27.427636 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:55:27.427832 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:55:27.428454 (+ 622us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 14:55:27.428456 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0927 14:55:27.428492 (+ 36us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 14:55:27.428631 (+ 139us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 14:55:27.428931 (+ 300us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 14:55:27.428937 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 14:55:27.428941 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 14:55:27.429040 (+ 99us) client_negotiation.cc:240] Negotiation successful
0927 14:55:27.429285 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":250,"thread_start_us":108,"threads_started":1}
I20260927 14:55:27.431047 18418 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33731
I20260927 14:55:27.431277 18418 heartbeater.cc:464] Registering TS with master...
I20260927 14:55:27.431746 18418 heartbeater.cc:511] Master 127.0.98.62:33731 requested a full tablet report, sending...
I20260927 14:55:27.432647 17968 ts_manager.cc:194] Registered new tserver with Master: 12b88a1e5a13473a907d6372bc7bc90f (127.0.98.3:39519)
I20260927 14:55:27.433167 17968 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.3:44635
I20260927 14:55:27.439260 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 14:55:27.444372 18007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:55:27.441201 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51300 (local address 127.0.98.62:33731)
0927 14:55:27.441318 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0927 14:55:27.441321 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 14:55:27.441497 (+ 176us) server_negotiation.cc:407] Connection header received
0927 14:55:27.441574 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:55:27.441576 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:55:27.441609 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:55:27.441670 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:55:27.441873 (+ 203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:55:27.442347 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:55:27.443112 (+ 765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:55:27.443311 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:55:27.443382 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:55:27.444040 (+ 658us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:55:27.444043 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:55:27.444045 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 14:55:27.444055 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:55:27.444071 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 14:55:27.444079 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:55:27.444133 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0927 14:55:27.444218 (+ 85us) server_negotiation.cc:299] Negotiation successful
0927 14:55:27.444277 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260927 14:55:27.446858 17968 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51300:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 14:55:27.449635 17968 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 14:55:27.463112 18429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:55:27.457194 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37635 (local address 127.0.0.1:35328)
0927 14:55:27.457772 (+ 578us) negotiation.cc:107] Waiting for socket to connect
0927 14:55:27.457818 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0927 14:55:27.457938 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:55:27.458852 (+ 914us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:55:27.458855 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:55:27.458873 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 14:55:27.458979 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:55:27.458990 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:55:27.460151 (+ 1161us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:55:27.460157 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:55:27.461835 (+ 1678us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:55:27.461840 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:55:27.462678 (+ 838us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:55:27.462681 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:55:27.462754 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:55:27.462770 (+ 16us) client_negotiation.cc:769] Sending connection context
0927 14:55:27.462814 (+ 44us) client_negotiation.cc:240] Negotiation successful
0927 14:55:27.462960 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":423,"mutex_wait_us":163,"thread_start_us":74,"threads_started":1}
I20260927 14:55:27.463181 18434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:55:27.458251 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43614 (local address 127.0.98.3:39519)
0927 14:55:27.458664 (+ 413us) server_negotiation.cc:206] Beginning negotiation
0927 14:55:27.458675 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 14:55:27.458696 (+ 21us) server_negotiation.cc:407] Connection header received
0927 14:55:27.458766 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:55:27.458774 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:55:27.458933 (+ 159us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:55:27.459074 (+ 141us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 14:55:27.459436 (+ 362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:55:27.460470 (+ 1034us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:55:27.461692 (+ 1222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:55:27.462405 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:55:27.462514 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:55:27.462701 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0927 14:55:27.462932 (+ 231us) server_negotiation.cc:299] Negotiation successful
0927 14:55:27.463031 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":335,"thread_start_us":146,"threads_started":1}
I20260927 14:55:27.463315 18430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:55:27.457281 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:43925 (local address 127.0.0.1:60700)
0927 14:55:27.458020 (+ 739us) negotiation.cc:107] Waiting for socket to connect
0927 14:55:27.458034 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 14:55:27.458126 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:55:27.458772 (+ 646us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:55:27.458777 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:55:27.458797 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 14:55:27.458929 (+ 132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:55:27.458935 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:55:27.460195 (+ 1260us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:55:27.460198 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:55:27.461828 (+ 1630us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:55:27.461840 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:55:27.462821 (+ 981us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:55:27.462825 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:55:27.462920 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:55:27.462943 (+ 23us) client_negotiation.cc:769] Sending connection context
0927 14:55:27.463005 (+ 62us) client_negotiation.cc:240] Negotiation successful
0927 14:55:27.463075 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":681,"mutex_wait_us":211,"thread_start_us":87,"threads_started":1}
I20260927 14:55:27.463672 18432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:55:27.457646 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60700 (local address 127.0.98.2:43925)
0927 14:55:27.458255 (+ 609us) server_negotiation.cc:206] Beginning negotiation
0927 14:55:27.458269 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 14:55:27.458296 (+ 27us) server_negotiation.cc:407] Connection header received
0927 14:55:27.458372 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:55:27.458377 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:55:27.458586 (+ 209us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:55:27.458772 (+ 186us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 14:55:27.459037 (+ 265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:55:27.460064 (+ 1027us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:55:27.461952 (+ 1888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:55:27.462695 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:55:27.462823 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:55:27.463006 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0927 14:55:27.463053 (+ 47us) server_negotiation.cc:299] Negotiation successful
0927 14:55:27.463141 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":494,"thread_start_us":129,"threads_started":1}
I20260927 14:55:27.464087 18431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:55:27.457322 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35328 (local address 127.0.98.1:37635)
0927 14:55:27.458371 (+ 1049us) server_negotiation.cc:206] Beginning negotiation
0927 14:55:27.458386 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 14:55:27.458412 (+ 26us) server_negotiation.cc:407] Connection header received
0927 14:55:27.458487 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:55:27.458492 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:55:27.458683 (+ 191us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:55:27.459114 (+ 431us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 14:55:27.459245 (+ 131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:55:27.460037 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:55:27.461981 (+ 1944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:55:27.462554 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:55:27.463595 (+ 1041us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:55:27.463755 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0927 14:55:27.463812 (+ 57us) server_negotiation.cc:299] Negotiation successful
0927 14:55:27.463907 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":946,"thread_start_us":109,"threads_started":1}
I20260927 14:55:27.464963 18433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:55:27.457868 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:39519 (local address 127.0.0.1:43614)
0927 14:55:27.458506 (+ 638us) negotiation.cc:107] Waiting for socket to connect
0927 14:55:27.458519 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 14:55:27.458596 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:55:27.459203 (+ 607us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:55:27.459205 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:55:27.459237 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 14:55:27.459313 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:55:27.459318 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:55:27.460596 (+ 1278us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:55:27.460601 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:55:27.461567 (+ 966us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:55:27.461576 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:55:27.462547 (+ 971us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:55:27.462550 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:55:27.462670 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:55:27.462713 (+ 43us) client_negotiation.cc:769] Sending connection context
0927 14:55:27.462830 (+ 117us) client_negotiation.cc:240] Negotiation successful
0927 14:55:27.464844 (+ 2014us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":582,"mutex_wait_us":1902,"thread_start_us":71,"threads_started":1}
I20260927 14:55:27.466275 18221 tablet_service.cc:1467] Processing CreateTablet for tablet 5c6b0c5ea7b741a0905639898fab0ca9 (DEFAULT_TABLE table=test-table [id=c03ecb2e689c4b29866e495f1e9b90ae]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 14:55:27.466743 18088 tablet_service.cc:1467] Processing CreateTablet for tablet 5c6b0c5ea7b741a0905639898fab0ca9 (DEFAULT_TABLE table=test-table [id=c03ecb2e689c4b29866e495f1e9b90ae]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 14:55:27.467247 18221 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5c6b0c5ea7b741a0905639898fab0ca9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 14:55:27.467708 18088 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5c6b0c5ea7b741a0905639898fab0ca9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 14:55:27.467408 18352 tablet_service.cc:1467] Processing CreateTablet for tablet 5c6b0c5ea7b741a0905639898fab0ca9 (DEFAULT_TABLE table=test-table [id=c03ecb2e689c4b29866e495f1e9b90ae]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 14:55:27.468322 18352 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5c6b0c5ea7b741a0905639898fab0ca9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 14:55:27.473177 18435 tablet_bootstrap.cc:492] T 5c6b0c5ea7b741a0905639898fab0ca9 P e043ff41b9ed40c4934257e6a098a0a9: Bootstrap starting.
I20260927 14:55:27.474771 18436 tablet_bootstrap.cc:492] T 5c6b0c5ea7b741a0905639898fab0ca9 P 12b88a1e5a13473a907d6372bc7bc90f: Bootstrap starting.
I20260927 14:55:27.475463 18437 tablet_bootstrap.cc:492] T 5c6b0c5ea7b741a0905639898fab0ca9 P ba5c9e0c0800457993170da02d0ad5bc: Bootstrap starting.
I20260927 14:55:27.476212 18435 tablet_bootstrap.cc:654] T 5c6b0c5ea7b741a0905639898fab0ca9 P e043ff41b9ed40c4934257e6a098a0a9: Neither blocks nor log segments found. Creating new log.
I20260927 14:55:27.477090 18435 log.cc:824] T 5c6b0c5ea7b741a0905639898fab0ca9 P e043ff41b9ed40c4934257e6a098a0a9: Log is configured to *not* fsync() on all Append() calls
I20260927 14:55:27.477553 18437 tablet_bootstrap.cc:654] T 5c6b0c5ea7b741a0905639898fab0ca9 P ba5c9e0c0800457993170da02d0ad5bc: Neither blocks nor log segments found. Creating new log.
I20260927 14:55:27.478353 18437 log.cc:824] T 5c6b0c5ea7b741a0905639898fab0ca9 P ba5c9e0c0800457993170da02d0ad5bc: Log is configured to *not* fsync() on all Append() calls
I20260927 14:55:27.478744 18436 tablet_bootstrap.cc:654] T 5c6b0c5ea7b741a0905639898fab0ca9 P 12b88a1e5a13473a907d6372bc7bc90f: Neither blocks nor log segments found. Creating new log.
I20260927 14:55:27.479184 18435 tablet_bootstrap.cc:492] T 5c6b0c5ea7b741a0905639898fab0ca9 P e043ff41b9ed40c4934257e6a098a0a9: No bootstrap required, opened a new log
I20260927 14:55:27.479368 18435 ts_tablet_manager.cc:1397] T 5c6b0c5ea7b741a0905639898fab0ca9 P e043ff41b9ed40c4934257e6a098a0a9: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260927 14:55:27.479518 18436 log.cc:824] T 5c6b0c5ea7b741a0905639898fab0ca9 P 12b88a1e5a13473a907d6372bc7bc90f: Log is configured to *not* fsync() on all Append() calls
I20260927 14:55:27.480947 18437 tablet_bootstrap.cc:492] T 5c6b0c5ea7b741a0905639898fab0ca9 P ba5c9e0c0800457993170da02d0ad5bc: No bootstrap required, opened a new log
I20260927 14:55:27.481070 18436 tablet_bootstrap.cc:492] T 5c6b0c5ea7b741a0905639898fab0ca9 P 12b88a1e5a13473a907d6372bc7bc90f: No bootstrap required, opened a new log
I20260927 14:55:27.481181 18437 ts_tablet_manager.cc:1397] T 5c6b0c5ea7b741a0905639898fab0ca9 P ba5c9e0c0800457993170da02d0ad5bc: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260927 14:55:27.481325 18436 ts_tablet_manager.cc:1397] T 5c6b0c5ea7b741a0905639898fab0ca9 P 12b88a1e5a13473a907d6372bc7bc90f: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260927 14:55:27.483654 18435 raft_consensus.cc:348] T 5c6b0c5ea7b741a0905639898fab0ca9 P e043ff41b9ed40c4934257e6a098a0a9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba5c9e0c0800457993170da02d0ad5bc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37635 } } peers { permanent_uuid: "e043ff41b9ed40c4934257e6a098a0a9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43925 } } peers { permanent_uuid: "12b88a1e5a13473a907d6372bc7bc90f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39519 } }
I20260927 14:55:27.483908 18435 raft_consensus.cc:374] T 5c6b0c5ea7b741a0905639898fab0ca9 P e043ff41b9ed40c4934257e6a098a0a9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 14:55:27.483963 18435 raft_consensus.cc:729] T 5c6b0c5ea7b741a0905639898fab0ca9 P e043ff41b9ed40c4934257e6a098a0a9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e043ff41b9ed40c4934257e6a098a0a9, State: Initialized, Role: FOLLOWER
I20260927 14:55:27.484252 18436 raft_consensus.cc:348] T 5c6b0c5ea7b741a0905639898fab0ca9 P 12b88a1e5a13473a907d6372bc7bc90f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba5c9e0c0800457993170da02d0ad5bc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37635 } } peers { permanent_uuid: "e043ff41b9ed40c4934257e6a098a0a9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43925 } } peers { permanent_uuid: "12b88a1e5a13473a907d6372bc7bc90f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39519 } }
I20260927 14:55:27.484470 18436 raft_consensus.cc:374] T 5c6b0c5ea7b741a0905639898fab0ca9 P 12b88a1e5a13473a907d6372bc7bc90f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 14:55:27.484412 18435 consensus_queue.cc:260] T 5c6b0c5ea7b741a0905639898fab0ca9 P e043ff41b9ed40c4934257e6a098a0a9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba5c9e0c0800457993170da02d0ad5bc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37635 } } peers { permanent_uuid: "e043ff41b9ed40c4934257e6a098a0a9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43925 } } peers { permanent_uuid: "12b88a1e5a13473a907d6372bc7bc90f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39519 } }
I20260927 14:55:27.484521 18436 raft_consensus.cc:729] T 5c6b0c5ea7b741a0905639898fab0ca9 P 12b88a1e5a13473a907d6372bc7bc90f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 12b88a1e5a13473a907d6372bc7bc90f, State: Initialized, Role: FOLLOWER
I20260927 14:55:27.484925 18436 consensus_queue.cc:260] T 5c6b0c5ea7b741a0905639898fab0ca9 P 12b88a1e5a13473a907d6372bc7bc90f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba5c9e0c0800457993170da02d0ad5bc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37635 } } peers { permanent_uuid: "e043ff41b9ed40c4934257e6a098a0a9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43925 } } peers { permanent_uuid: "12b88a1e5a13473a907d6372bc7bc90f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39519 } }
I20260927 14:55:27.484855 18437 raft_consensus.cc:348] T 5c6b0c5ea7b741a0905639898fab0ca9 P ba5c9e0c0800457993170da02d0ad5bc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba5c9e0c0800457993170da02d0ad5bc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37635 } } peers { permanent_uuid: "e043ff41b9ed40c4934257e6a098a0a9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43925 } } peers { permanent_uuid: "12b88a1e5a13473a907d6372bc7bc90f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39519 } }
I20260927 14:55:27.485097 18437 raft_consensus.cc:374] T 5c6b0c5ea7b741a0905639898fab0ca9 P ba5c9e0c0800457993170da02d0ad5bc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 14:55:27.485129 18287 heartbeater.cc:503] Master 127.0.98.62:33731 was elected leader, sending a full tablet report...
I20260927 14:55:27.485153 18437 raft_consensus.cc:729] T 5c6b0c5ea7b741a0905639898fab0ca9 P ba5c9e0c0800457993170da02d0ad5bc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ba5c9e0c0800457993170da02d0ad5bc, State: Initialized, Role: FOLLOWER
I20260927 14:55:27.485401 18435 ts_tablet_manager.cc:1428] T 5c6b0c5ea7b741a0905639898fab0ca9 P e043ff41b9ed40c4934257e6a098a0a9: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260927 14:55:27.485551 18437 consensus_queue.cc:260] T 5c6b0c5ea7b741a0905639898fab0ca9 P ba5c9e0c0800457993170da02d0ad5bc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba5c9e0c0800457993170da02d0ad5bc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37635 } } peers { permanent_uuid: "e043ff41b9ed40c4934257e6a098a0a9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43925 } } peers { permanent_uuid: "12b88a1e5a13473a907d6372bc7bc90f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39519 } }
W20260927 14:55:27.485718 18288 tablet.cc:2369] T 5c6b0c5ea7b741a0905639898fab0ca9 P e043ff41b9ed40c4934257e6a098a0a9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 14:55:27.485795 18436 ts_tablet_manager.cc:1428] T 5c6b0c5ea7b741a0905639898fab0ca9 P 12b88a1e5a13473a907d6372bc7bc90f: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20260927 14:55:27.486285 18418 heartbeater.cc:503] Master 127.0.98.62:33731 was elected leader, sending a full tablet report...
I20260927 14:55:27.486406 18437 ts_tablet_manager.cc:1428] T 5c6b0c5ea7b741a0905639898fab0ca9 P ba5c9e0c0800457993170da02d0ad5bc: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260927 14:55:27.486696 18156 heartbeater.cc:503] Master 127.0.98.62:33731 was elected leader, sending a full tablet report...
W20260927 14:55:27.539019 18157 tablet.cc:2369] T 5c6b0c5ea7b741a0905639898fab0ca9 P ba5c9e0c0800457993170da02d0ad5bc: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 14:55:27.596773 18442 raft_consensus.cc:482] T 5c6b0c5ea7b741a0905639898fab0ca9 P 12b88a1e5a13473a907d6372bc7bc90f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 14:55:27.596956 18442 raft_consensus.cc:504] T 5c6b0c5ea7b741a0905639898fab0ca9 P 12b88a1e5a13473a907d6372bc7bc90f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba5c9e0c0800457993170da02d0ad5bc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37635 } } peers { permanent_uuid: "e043ff41b9ed40c4934257e6a098a0a9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43925 } } peers { permanent_uuid: "12b88a1e5a13473a907d6372bc7bc90f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39519 } }
I20260927 14:55:27.598011 18442 leader_election.cc:288] T 5c6b0c5ea7b741a0905639898fab0ca9 P 12b88a1e5a13473a907d6372bc7bc90f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ba5c9e0c0800457993170da02d0ad5bc (127.0.98.1:37635), e043ff41b9ed40c4934257e6a098a0a9 (127.0.98.2:43925)
I20260927 14:55:27.601680 18431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:55:27.598369 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:37925 (local address 127.0.98.1:37635)
0927 14:55:27.598487 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0927 14:55:27.598493 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 14:55:27.598513 (+ 20us) server_negotiation.cc:407] Connection header received
0927 14:55:27.598569 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:55:27.598572 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:55:27.598638 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:55:27.598718 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 14:55:27.598991 (+ 273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:55:27.599681 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:55:27.600730 (+ 1049us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:55:27.601150 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:55:27.601267 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:55:27.601325 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0927 14:55:27.601492 (+ 167us) server_negotiation.cc:299] Negotiation successful
0927 14:55:27.601551 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260927 14:55:27.601680 18420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:55:27.598250 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37635 (local address 127.0.98.3:37925)
0927 14:55:27.598356 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0927 14:55:27.598372 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 14:55:27.598465 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:55:27.598747 (+ 282us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:55:27.598749 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:55:27.598769 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 14:55:27.598867 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:55:27.598872 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:55:27.599782 (+ 910us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:55:27.599786 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:55:27.600609 (+ 823us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:55:27.600618 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:55:27.601264 (+ 646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:55:27.601268 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:55:27.601364 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:55:27.601385 (+ 21us) client_negotiation.cc:769] Sending connection context
0927 14:55:27.601479 (+ 94us) client_negotiation.cc:240] Negotiation successful
0927 14:55:27.601546 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31}
I20260927 14:55:27.602047 18432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:55:27.598369 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:58853 (local address 127.0.98.2:43925)
0927 14:55:27.598507 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0927 14:55:27.598511 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 14:55:27.599057 (+ 546us) server_negotiation.cc:407] Connection header received
0927 14:55:27.599154 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:55:27.599156 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:55:27.599200 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:55:27.599262 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 14:55:27.599522 (+ 260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:55:27.599978 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:55:27.600991 (+ 1013us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:55:27.601434 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:55:27.601551 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:55:27.601605 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0927 14:55:27.601858 (+ 253us) server_negotiation.cc:299] Negotiation successful
0927 14:55:27.601930 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260927 14:55:27.602365 18444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:55:27.598250 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:43925 (local address 127.0.98.3:58853)
0927 14:55:27.598960 (+ 710us) negotiation.cc:107] Waiting for socket to connect
0927 14:55:27.598974 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 14:55:27.599070 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:55:27.599291 (+ 221us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:55:27.599294 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:55:27.599311 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 14:55:27.599422 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:55:27.599427 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:55:27.600088 (+ 661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:55:27.600094 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:55:27.600862 (+ 768us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:55:27.600870 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:55:27.601570 (+ 700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:55:27.601574 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:55:27.601673 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:55:27.601701 (+ 28us) client_negotiation.cc:769] Sending connection context
0927 14:55:27.602185 (+ 484us) client_negotiation.cc:240] Negotiation successful
0927 14:55:27.602260 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":631,"mutex_wait_us":286,"thread_start_us":117,"threads_started":1}
I20260927 14:55:27.602315 18110 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5c6b0c5ea7b741a0905639898fab0ca9" candidate_uuid: "12b88a1e5a13473a907d6372bc7bc90f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ba5c9e0c0800457993170da02d0ad5bc" is_pre_election: true
I20260927 14:55:27.602631 18110 raft_consensus.cc:2454] T 5c6b0c5ea7b741a0905639898fab0ca9 P ba5c9e0c0800457993170da02d0ad5bc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 12b88a1e5a13473a907d6372bc7bc90f in term 0.
I20260927 14:55:27.602924 18241 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5c6b0c5ea7b741a0905639898fab0ca9" candidate_uuid: "12b88a1e5a13473a907d6372bc7bc90f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e043ff41b9ed40c4934257e6a098a0a9" is_pre_election: true
I20260927 14:55:27.603118 18308 leader_election.cc:302] T 5c6b0c5ea7b741a0905639898fab0ca9 P 12b88a1e5a13473a907d6372bc7bc90f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 12b88a1e5a13473a907d6372bc7bc90f, ba5c9e0c0800457993170da02d0ad5bc; no voters:
I20260927 14:55:27.603173 18241 raft_consensus.cc:2454] T 5c6b0c5ea7b741a0905639898fab0ca9 P e043ff41b9ed40c4934257e6a098a0a9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 12b88a1e5a13473a907d6372bc7bc90f in term 0.
I20260927 14:55:27.603401 18442 raft_consensus.cc:2781] T 5c6b0c5ea7b741a0905639898fab0ca9 P 12b88a1e5a13473a907d6372bc7bc90f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 14:55:27.603498 18442 raft_consensus.cc:482] T 5c6b0c5ea7b741a0905639898fab0ca9 P 12b88a1e5a13473a907d6372bc7bc90f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 14:55:27.603557 18442 raft_consensus.cc:3037] T 5c6b0c5ea7b741a0905639898fab0ca9 P 12b88a1e5a13473a907d6372bc7bc90f [term 0 FOLLOWER]: Advancing to term 1
I20260927 14:55:27.604553 18442 raft_consensus.cc:504] T 5c6b0c5ea7b741a0905639898fab0ca9 P 12b88a1e5a13473a907d6372bc7bc90f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba5c9e0c0800457993170da02d0ad5bc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37635 } } peers { permanent_uuid: "e043ff41b9ed40c4934257e6a098a0a9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43925 } } peers { permanent_uuid: "12b88a1e5a13473a907d6372bc7bc90f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39519 } }
I20260927 14:55:27.604894 18442 leader_election.cc:288] T 5c6b0c5ea7b741a0905639898fab0ca9 P 12b88a1e5a13473a907d6372bc7bc90f [CANDIDATE]: Term 1 election: Requested vote from peers ba5c9e0c0800457993170da02d0ad5bc (127.0.98.1:37635), e043ff41b9ed40c4934257e6a098a0a9 (127.0.98.2:43925)
I20260927 14:55:27.605253 18110 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5c6b0c5ea7b741a0905639898fab0ca9" candidate_uuid: "12b88a1e5a13473a907d6372bc7bc90f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ba5c9e0c0800457993170da02d0ad5bc"
I20260927 14:55:27.605281 18241 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5c6b0c5ea7b741a0905639898fab0ca9" candidate_uuid: "12b88a1e5a13473a907d6372bc7bc90f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e043ff41b9ed40c4934257e6a098a0a9"
I20260927 14:55:27.605378 18241 raft_consensus.cc:3037] T 5c6b0c5ea7b741a0905639898fab0ca9 P e043ff41b9ed40c4934257e6a098a0a9 [term 0 FOLLOWER]: Advancing to term 1
I20260927 14:55:27.605379 18110 raft_consensus.cc:3037] T 5c6b0c5ea7b741a0905639898fab0ca9 P ba5c9e0c0800457993170da02d0ad5bc [term 0 FOLLOWER]: Advancing to term 1
I20260927 14:55:27.606639 18110 raft_consensus.cc:2454] T 5c6b0c5ea7b741a0905639898fab0ca9 P ba5c9e0c0800457993170da02d0ad5bc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 12b88a1e5a13473a907d6372bc7bc90f in term 1.
I20260927 14:55:27.606639 18241 raft_consensus.cc:2454] T 5c6b0c5ea7b741a0905639898fab0ca9 P e043ff41b9ed40c4934257e6a098a0a9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 12b88a1e5a13473a907d6372bc7bc90f in term 1.
I20260927 14:55:27.606930 18307 leader_election.cc:302] T 5c6b0c5ea7b741a0905639898fab0ca9 P 12b88a1e5a13473a907d6372bc7bc90f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 12b88a1e5a13473a907d6372bc7bc90f, e043ff41b9ed40c4934257e6a098a0a9; no voters:
I20260927 14:55:27.607093 18442 raft_consensus.cc:2781] T 5c6b0c5ea7b741a0905639898fab0ca9 P 12b88a1e5a13473a907d6372bc7bc90f [term 1 FOLLOWER]: Leader election won for term 1
I20260927 14:55:27.607326 18442 raft_consensus.cc:686] T 5c6b0c5ea7b741a0905639898fab0ca9 P 12b88a1e5a13473a907d6372bc7bc90f [term 1 LEADER]: Becoming Leader. State: Replica: 12b88a1e5a13473a907d6372bc7bc90f, State: Running, Role: LEADER
I20260927 14:55:27.607659 18442 consensus_queue.cc:237] T 5c6b0c5ea7b741a0905639898fab0ca9 P 12b88a1e5a13473a907d6372bc7bc90f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba5c9e0c0800457993170da02d0ad5bc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37635 } } peers { permanent_uuid: "e043ff41b9ed40c4934257e6a098a0a9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43925 } } peers { permanent_uuid: "12b88a1e5a13473a907d6372bc7bc90f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39519 } }
I20260927 14:55:27.610733 17968 catalog_manager.cc:5495] T 5c6b0c5ea7b741a0905639898fab0ca9 P 12b88a1e5a13473a907d6372bc7bc90f reported cstate change: term changed from 0 to 1, leader changed from <none> to 12b88a1e5a13473a907d6372bc7bc90f (127.0.98.3). New cstate: current_term: 1 leader_uuid: "12b88a1e5a13473a907d6372bc7bc90f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba5c9e0c0800457993170da02d0ad5bc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37635 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e043ff41b9ed40c4934257e6a098a0a9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43925 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "12b88a1e5a13473a907d6372bc7bc90f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39519 } health_report { overall_health: HEALTHY } } }
I20260927 14:55:27.651144 18434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:55:27.647989 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43628 (local address 127.0.98.3:39519)
0927 14:55:27.648121 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0927 14:55:27.648125 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 14:55:27.648137 (+ 12us) server_negotiation.cc:407] Connection header received
0927 14:55:27.648210 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:55:27.648212 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:55:27.648272 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:55:27.648359 (+ 87us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 14:55:27.648562 (+ 203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:55:27.649044 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:55:27.649872 (+ 828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:55:27.650071 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:55:27.650144 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:55:27.650412 (+ 268us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 14:55:27.650826 (+ 414us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 14:55:27.650914 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0927 14:55:27.650979 (+ 65us) server_negotiation.cc:299] Negotiation successful
0927 14:55:27.651022 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260927 14:55:27.656738 18241 raft_consensus.cc:1261] T 5c6b0c5ea7b741a0905639898fab0ca9 P e043ff41b9ed40c4934257e6a098a0a9 [term 1 FOLLOWER]: Refusing update from remote peer 12b88a1e5a13473a907d6372bc7bc90f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 14:55:27.656765 18110 raft_consensus.cc:1261] T 5c6b0c5ea7b741a0905639898fab0ca9 P ba5c9e0c0800457993170da02d0ad5bc [term 1 FOLLOWER]: Refusing update from remote peer 12b88a1e5a13473a907d6372bc7bc90f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 14:55:27.657498 18445 consensus_queue.cc:1035] T 5c6b0c5ea7b741a0905639898fab0ca9 P 12b88a1e5a13473a907d6372bc7bc90f [LEADER]: Connected to new peer: Peer: permanent_uuid: "ba5c9e0c0800457993170da02d0ad5bc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37635 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 14:55:27.657677 18442 consensus_queue.cc:1035] T 5c6b0c5ea7b741a0905639898fab0ca9 P 12b88a1e5a13473a907d6372bc7bc90f [LEADER]: Connected to new peer: Peer: permanent_uuid: "e043ff41b9ed40c4934257e6a098a0a9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43925 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 14:55:27.664933 18448 mvcc.cc:205] Tried to move back new op lower bound from 7333973719674777600 to 7333973719483600896. Current Snapshot: MvccSnapshot[applied={T|T < 7333973719674777600}]
I20260927 14:55:27.666276 18449 mvcc.cc:205] Tried to move back new op lower bound from 7333973719674777600 to 7333973719483600896. Current Snapshot: MvccSnapshot[applied={T|T < 7333973719674777600}]
I20260927 14:55:27.673345 17968 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51300:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 14:55:27.673557 17968 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51300:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 14:55:27.675800 17968 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 3cc5b5bd948c4f6ead0c0f1287d40cd1: Sending DeleteTablet for 3 replicas of tablet 5c6b0c5ea7b741a0905639898fab0ca9
I20260927 14:55:27.676635 18088 tablet_service.cc:1514] Processing DeleteTablet for tablet 5c6b0c5ea7b741a0905639898fab0ca9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 14:55:27 UTC) from {username='slave'} at 127.0.0.1:35328
I20260927 14:55:27.676651 18352 tablet_service.cc:1514] Processing DeleteTablet for tablet 5c6b0c5ea7b741a0905639898fab0ca9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 14:55:27 UTC) from {username='slave'} at 127.0.0.1:43614
I20260927 14:55:27.676654 18221 tablet_service.cc:1514] Processing DeleteTablet for tablet 5c6b0c5ea7b741a0905639898fab0ca9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 14:55:27 UTC) from {username='slave'} at 127.0.0.1:60700
I20260927 14:55:27.677199 18459 tablet_replica.cc:330] T 5c6b0c5ea7b741a0905639898fab0ca9 P e043ff41b9ed40c4934257e6a098a0a9: stopping tablet replica
I20260927 14:55:27.677580 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6936m5/build/debug/bin/kudu with pid 18028
I20260927 14:55:27.677582 18459 raft_consensus.cc:2229] T 5c6b0c5ea7b741a0905639898fab0ca9 P e043ff41b9ed40c4934257e6a098a0a9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 14:55:27.678767 18459 raft_consensus.cc:2258] T 5c6b0c5ea7b741a0905639898fab0ca9 P e043ff41b9ed40c4934257e6a098a0a9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 14:55:27.679500 18457 tablet_replica.cc:330] T 5c6b0c5ea7b741a0905639898fab0ca9 P 12b88a1e5a13473a907d6372bc7bc90f: stopping tablet replica
I20260927 14:55:27.679770 18457 raft_consensus.cc:2229] T 5c6b0c5ea7b741a0905639898fab0ca9 P 12b88a1e5a13473a907d6372bc7bc90f [term 1 LEADER]: Raft consensus shutting down.
I20260927 14:55:27.680109 18457 raft_consensus.cc:2258] T 5c6b0c5ea7b741a0905639898fab0ca9 P 12b88a1e5a13473a907d6372bc7bc90f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 14:55:27.680365 18459 ts_tablet_manager.cc:1905] T 5c6b0c5ea7b741a0905639898fab0ca9 P e043ff41b9ed40c4934257e6a098a0a9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 14:55:27.681419 18457 ts_tablet_manager.cc:1905] T 5c6b0c5ea7b741a0905639898fab0ca9 P 12b88a1e5a13473a907d6372bc7bc90f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 14:55:27.684064 18457 ts_tablet_manager.cc:1918] T 5c6b0c5ea7b741a0905639898fab0ca9 P 12b88a1e5a13473a907d6372bc7bc90f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 14:55:27.684190 18459 ts_tablet_manager.cc:1918] T 5c6b0c5ea7b741a0905639898fab0ca9 P e043ff41b9ed40c4934257e6a098a0a9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 14:55:27.684257 18457 log.cc:1196] T 5c6b0c5ea7b741a0905639898fab0ca9 P 12b88a1e5a13473a907d6372bc7bc90f: Deleting WAL directory at /tmp/dist-test-task6936m5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790520684602251-392-0/minicluster-data/ts-2/wal/wals/5c6b0c5ea7b741a0905639898fab0ca9
I20260927 14:55:27.684317 18459 log.cc:1196] T 5c6b0c5ea7b741a0905639898fab0ca9 P e043ff41b9ed40c4934257e6a098a0a9: Deleting WAL directory at /tmp/dist-test-task6936m5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790520684602251-392-0/minicluster-data/ts-1/wal/wals/5c6b0c5ea7b741a0905639898fab0ca9
I20260927 14:55:27.684675 18459 ts_tablet_manager.cc:1939] T 5c6b0c5ea7b741a0905639898fab0ca9 P e043ff41b9ed40c4934257e6a098a0a9: Deleting consensus metadata
I20260927 14:55:27.684710 18457 ts_tablet_manager.cc:1939] T 5c6b0c5ea7b741a0905639898fab0ca9 P 12b88a1e5a13473a907d6372bc7bc90f: Deleting consensus metadata
W20260927 14:55:27.685783 17955 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:37635 (error 108)
I20260927 14:55:27.685971 17954 catalog_manager.cc:4841] TS e043ff41b9ed40c4934257e6a098a0a9 (127.0.98.2:43925): tablet 5c6b0c5ea7b741a0905639898fab0ca9 (table test-table [id=c03ecb2e689c4b29866e495f1e9b90ae]) successfully deleted
I20260927 14:55:27.686189 17954 catalog_manager.cc:4841] TS 12b88a1e5a13473a907d6372bc7bc90f (127.0.98.3:39519): tablet 5c6b0c5ea7b741a0905639898fab0ca9 (table test-table [id=c03ecb2e689c4b29866e495f1e9b90ae]) successfully deleted
I20260927 14:55:27.686651 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6936m5/build/debug/bin/kudu with pid 18159
I20260927 14:55:27.686846 18433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:55:27.686541 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37635 (local address 127.0.0.1:35342)
0927 14:55:27.686669 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0927 14:55:27.686747 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:37635: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260927 14:55:27.687605 17955 catalog_manager.cc:4584] TS ba5c9e0c0800457993170da02d0ad5bc (127.0.98.1:37635): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 5c6b0c5ea7b741a0905639898fab0ca9: Network error: Client connection negotiation failed: client connection to 127.0.98.1:37635: connect: Connection refused (error 111)
I20260927 14:55:27.693893 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6936m5/build/debug/bin/kudu with pid 18290
I20260927 14:55:27.699393 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6936m5/build/debug/bin/kudu with pid 17937
2026-09-27T14:55:27Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (1082 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-task6936m5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790520684602251-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 14:55:27 dist-test-slave-xk1g krb5kdc[18463](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 14:55:27 dist-test-slave-xk1g krb5kdc[18463](info): set up 2 sockets
Sep 27 14:55:27 dist-test-slave-xk1g krb5kdc[18463](info): commencing operation
krb5kdc: starting...
W20260927 14:55:28.728829 392 mini_kdc.cc:121] Time spent starting KDC: real 1.008s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6936m5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790520684602251-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6936m5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790520684602251-392-0/krb5kdc/test-user.keytab.
Sep 27 14:55:28 dist-test-slave-xk1g krb5kdc[18463](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790520928, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T14:55:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T14:55:28Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6936m5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790520684602251-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6936m5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790520684602251-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 14:55:28.852968 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6936m5/build/debug/bin/kudu
/tmp/dist-test-task6936m5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790520684602251-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790520684602251-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790520684602251-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790520684602251-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39531
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790520684602251-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33327
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:39531
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790520684602251-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790520684602251-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790520684602251-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 14:55:28.964448 18476 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 14:55:28.964735 18476 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 14:55:28.964776 18476 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 14:55:28.968781 18476 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 14:55:28.968837 18476 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 14:55:28.968856 18476 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 14:55:28.968873 18476 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 14:55:28.968887 18476 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 14:55:28.974102 18476 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33327
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790520684602251-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790520684602251-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:39531
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790520684602251-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39531
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790520684602251-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18476
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790520684602251-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:15:17 UTC on ccycloud.quasar-ledxfe.root.comops.site
build id 4676134
I20260927 14:55:28.975143 18476 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 14:55:28.976158 18476 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 14:55:28.984582 18482 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 14:55:28.984596 18484 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 14:55:28.984622 18481 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 14:55:28.984818 18476 server_base.cc:1034] running on GCE node
I20260927 14:55:28.985484 18476 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 14:55:28.986575 18476 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 14:55:28.987792 18476 hybrid_clock.cc:648] HybridClock initialized: now 1790520928987777 us; error 28 us; skew 500 ppm
Sep 27 14:55:28 dist-test-slave-xk1g krb5kdc[18463](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790520928, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 14:55:28.990747 18476 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 14:55:28.991600 18476 webserver.cc:460] Webserver started at http://127.0.98.62:42125/ using document root <none> and password file <none>
I20260927 14:55:28.992125 18476 fs_manager.cc:359] Metadata directory not provided
I20260927 14:55:28.992183 18476 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 14:55:28.992348 18476 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 14:55:28.994424 18476 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6936m5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790520684602251-392-0/minicluster-data/master-0/data/instance:
uuid: "5184c6a24d254960ae0f5ff3006e82af"
format_stamp: "Formatted at 2026-09-27 14:55:28 on dist-test-slave-xk1g"
server_key: "cc7c80a16bb31adc3ca1f417b5465472"
server_key_iv: "3755fb43a1735d5a4df7e77fa3ce0714"
server_key_version: "encryptionkey@0"
I20260927 14:55:28.994947 18476 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6936m5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790520684602251-392-0/minicluster-data/master-0/wal/instance:
uuid: "5184c6a24d254960ae0f5ff3006e82af"
format_stamp: "Formatted at 2026-09-27 14:55:28 on dist-test-slave-xk1g"
server_key: "cc7c80a16bb31adc3ca1f417b5465472"
server_key_iv: "3755fb43a1735d5a4df7e77fa3ce0714"
server_key_version: "encryptionkey@0"
I20260927 14:55:28.998692 18476 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260927 14:55:29.001206 18491 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 14:55:29.002488 18476 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 14:55:29.002624 18476 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790520684602251-392-0/minicluster-data/master-0/data,/tmp/dist-test-task6936m5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790520684602251-392-0/minicluster-data/master-0/wal
uuid: "5184c6a24d254960ae0f5ff3006e82af"
format_stamp: "Formatted at 2026-09-27 14:55:28 on dist-test-slave-xk1g"
server_key: "cc7c80a16bb31adc3ca1f417b5465472"
server_key_iv: "3755fb43a1735d5a4df7e77fa3ce0714"
server_key_version: "encryptionkey@0"
I20260927 14:55:29.002749 18476 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790520684602251-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790520684602251-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790520684602251-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 14:55:29.014210 18476 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 14:55:29.017448 18476 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 14:55:29.017654 18476 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 14:55:29.024652 18476 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:39531
I20260927 14:55:29.024720 18543 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:39531 every 8 connection(s)
I20260927 14:55:29.025682 18476 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6936m5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790520684602251-392-0/minicluster-data/master-0/data/info.pb
I20260927 14:55:29.028414 18544 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 14:55:29.030319 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6936m5/build/debug/bin/kudu as pid 18476
I20260927 14:55:29.030424 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6936m5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790520684602251-392-0/minicluster-data/master-0/wal/instance
I20260927 14:55:29.030689 392 external_mini_cluster.cc:1428] Setting key e656aa8b419930f6168bde3d9f6c7e58
I20260927 14:55:29.034240 18544 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5184c6a24d254960ae0f5ff3006e82af: Bootstrap starting.
I20260927 14:55:29.036350 18544 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5184c6a24d254960ae0f5ff3006e82af: Neither blocks nor log segments found. Creating new log.
I20260927 14:55:29.037123 18544 log.cc:824] T 00000000000000000000000000000000 P 5184c6a24d254960ae0f5ff3006e82af: Log is configured to *not* fsync() on all Append() calls
I20260927 14:55:29.038901 18544 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5184c6a24d254960ae0f5ff3006e82af: No bootstrap required, opened a new log
Sep 27 14:55:29 dist-test-slave-xk1g krb5kdc[18463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790520928, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 14:55:29.042250 18544 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5184c6a24d254960ae0f5ff3006e82af [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5184c6a24d254960ae0f5ff3006e82af" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39531 } }
I20260927 14:55:29.042495 18544 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5184c6a24d254960ae0f5ff3006e82af [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 14:55:29.042575 18544 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5184c6a24d254960ae0f5ff3006e82af [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5184c6a24d254960ae0f5ff3006e82af, State: Initialized, Role: FOLLOWER
I20260927 14:55:29.043052 18544 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5184c6a24d254960ae0f5ff3006e82af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5184c6a24d254960ae0f5ff3006e82af" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39531 } }
I20260927 14:55:29.043196 18544 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5184c6a24d254960ae0f5ff3006e82af [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 14:55:29.043267 18544 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5184c6a24d254960ae0f5ff3006e82af [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 14:55:29.043382 18544 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5184c6a24d254960ae0f5ff3006e82af [term 0 FOLLOWER]: Advancing to term 1
I20260927 14:55:29.044240 18544 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5184c6a24d254960ae0f5ff3006e82af [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5184c6a24d254960ae0f5ff3006e82af" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39531 } }
I20260927 14:55:29.044598 18544 leader_election.cc:302] T 00000000000000000000000000000000 P 5184c6a24d254960ae0f5ff3006e82af [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5184c6a24d254960ae0f5ff3006e82af; no voters:
I20260927 14:55:29.044902 18544 leader_election.cc:288] T 00000000000000000000000000000000 P 5184c6a24d254960ae0f5ff3006e82af [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 14:55:29.044993 18549 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5184c6a24d254960ae0f5ff3006e82af [term 1 FOLLOWER]: Leader election won for term 1
I20260927 14:55:29.045285 18549 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5184c6a24d254960ae0f5ff3006e82af [term 1 LEADER]: Becoming Leader. State: Replica: 5184c6a24d254960ae0f5ff3006e82af, State: Running, Role: LEADER
I20260927 14:55:29.045576 18549 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5184c6a24d254960ae0f5ff3006e82af [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5184c6a24d254960ae0f5ff3006e82af" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39531 } }
I20260927 14:55:29.045707 18547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:55:29.032056 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49380 (local address 127.0.98.62:39531)
0927 14:55:29.032485 (+ 429us) server_negotiation.cc:206] Beginning negotiation
0927 14:55:29.032499 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 14:55:29.032532 (+ 33us) server_negotiation.cc:407] Connection header received
0927 14:55:29.033410 (+ 878us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:55:29.033428 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:55:29.033751 (+ 323us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:55:29.034084 (+ 333us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:55:29.034887 (+ 803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:55:29.035860 (+ 973us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:55:29.036795 (+ 935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:55:29.037366 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:55:29.037532 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:55:29.039982 (+ 2450us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:55:29.039995 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:55:29.040009 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 14:55:29.040036 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:55:29.042337 (+ 2301us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:55:29.042835 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:55:29.042840 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:55:29.042845 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:55:29.042953 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:55:29.043307 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:55:29.043310 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:55:29.043311 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:55:29.043782 (+ 471us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:55:29.043992 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0927 14:55:29.044299 (+ 307us) server_negotiation.cc:299] Negotiation successful
0927 14:55:29.044619 (+ 320us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":296,"thread_start_us":129,"threads_started":1}
I20260927 14:55:29.046192 18544 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5184c6a24d254960ae0f5ff3006e82af [sys.catalog]: configured and running, proceeding with master startup.
I20260927 14:55:29.046751 18549 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5184c6a24d254960ae0f5ff3006e82af [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5184c6a24d254960ae0f5ff3006e82af. Latest consensus state: current_term: 1 leader_uuid: "5184c6a24d254960ae0f5ff3006e82af" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5184c6a24d254960ae0f5ff3006e82af" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39531 } } }
I20260927 14:55:29.046859 18549 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5184c6a24d254960ae0f5ff3006e82af [sys.catalog]: This master's current role is: LEADER
I20260927 14:55:29.047176 18551 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5184c6a24d254960ae0f5ff3006e82af [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5184c6a24d254960ae0f5ff3006e82af" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5184c6a24d254960ae0f5ff3006e82af" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39531 } } }
I20260927 14:55:29.047276 18551 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5184c6a24d254960ae0f5ff3006e82af [sys.catalog]: This master's current role is: LEADER
W20260927 14:55:29.051303 18564 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 5184c6a24d254960ae0f5ff3006e82af: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 14:55:29.051383 18564 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 14:55:29.051437 18554 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 14:55:29.052268 18554 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 14:55:29.057368 18554 catalog_manager.cc:1347] Generated new cluster ID: ed90be9c0aa241f1bbdc9fc84c813499
I20260927 14:55:29.057439 18554 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 14:55:29.068015 18554 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 14:55:29.069286 18554 catalog_manager.cc:1504] Loading token signing keys...
I20260927 14:55:29.077533 18554 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5184c6a24d254960ae0f5ff3006e82af: Generated new TSK 0
I20260927 14:55:29.078418 18554 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6936m5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790520684602251-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6936m5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790520684602251-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260927 14:55:29.115876 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6936m5/build/debug/bin/kudu
/tmp/dist-test-task6936m5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790520684602251-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790520684602251-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790520684602251-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790520684602251-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39531
--keytab_file=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790520684602251-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33327
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790520684602251-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790520684602251-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790520684602251-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 14:55:29.225195 18570 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 14:55:29.225425 18570 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 14:55:29.225466 18570 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 14:55:29.229405 18570 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 14:55:29.229455 18570 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 14:55:29.229508 18570 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 14:55:29.234560 18570 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33327
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790520684602251-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790520684602251-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790520684602251-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790520684602251-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39531
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18570
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790520684602251-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:15:17 UTC on ccycloud.quasar-ledxfe.root.comops.site
build id 4676134
I20260927 14:55:29.235477 18570 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 14:55:29.236397 18570 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 14:55:29.245548 18578 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 14:55:29.245570 18576 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 14:55:29.245617 18575 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 14:55:29.245745 18570 server_base.cc:1034] running on GCE node
I20260927 14:55:29.246243 18570 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 14:55:29.246901 18570 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 14:55:29.248108 18570 hybrid_clock.cc:648] HybridClock initialized: now 1790520929248091 us; error 33 us; skew 500 ppm
Sep 27 14:55:29 dist-test-slave-xk1g krb5kdc[18463](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790520929, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 14:55:29.250681 18570 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 14:55:29.251494 18570 webserver.cc:460] Webserver started at http://127.0.98.1:43369/ using document root <none> and password file <none>
I20260927 14:55:29.252046 18570 fs_manager.cc:359] Metadata directory not provided
I20260927 14:55:29.252137 18570 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 14:55:29.252370 18570 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 14:55:29.254385 18570 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6936m5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790520684602251-392-0/minicluster-data/ts-0/data/instance:
uuid: "a9045ffd246740d1b1d8f42525689314"
format_stamp: "Formatted at 2026-09-27 14:55:29 on dist-test-slave-xk1g"
server_key: "44bb85b48dbbc1833aad43bf092a6b3c"
server_key_iv: "97bde0f71e3f96a7193b0769207578c2"
server_key_version: "encryptionkey@0"
I20260927 14:55:29.254933 18570 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6936m5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790520684602251-392-0/minicluster-data/ts-0/wal/instance:
uuid: "a9045ffd246740d1b1d8f42525689314"
format_stamp: "Formatted at 2026-09-27 14:55:29 on dist-test-slave-xk1g"
server_key: "44bb85b48dbbc1833aad43bf092a6b3c"
server_key_iv: "97bde0f71e3f96a7193b0769207578c2"
server_key_version: "encryptionkey@0"
I20260927 14:55:29.258461 18570 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 14:55:29.260692 18585 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 14:55:29.261865 18570 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 14:55:29.261997 18570 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790520684602251-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task6936m5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790520684602251-392-0/minicluster-data/ts-0/wal
uuid: "a9045ffd246740d1b1d8f42525689314"
format_stamp: "Formatted at 2026-09-27 14:55:29 on dist-test-slave-xk1g"
server_key: "44bb85b48dbbc1833aad43bf092a6b3c"
server_key_iv: "97bde0f71e3f96a7193b0769207578c2"
server_key_version: "encryptionkey@0"
I20260927 14:55:29.262108 18570 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790520684602251-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790520684602251-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790520684602251-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 14:55:29.294291 18570 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 14:55:29.297458 18570 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 14:55:29.297662 18570 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 14:55:29.298254 18570 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 14:55:29.299067 18570 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 14:55:29.299139 18570 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 14:55:29.299211 18570 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 14:55:29.299271 18570 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 14:55:29.308283 18570 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:37571
I20260927 14:55:29.308301 18698 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:37571 every 8 connection(s)
I20260927 14:55:29.309271 18570 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6936m5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790520684602251-392-0/minicluster-data/ts-0/data/info.pb
I20260927 14:55:29.312206 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6936m5/build/debug/bin/kudu as pid 18570
I20260927 14:55:29.312337 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6936m5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790520684602251-392-0/minicluster-data/ts-0/wal/instance
I20260927 14:55:29.312623 392 external_mini_cluster.cc:1428] Setting key 6e91af9ea791eba91087699523004116
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Sep 27 14:55:29 dist-test-slave-xk1g krb5kdc[18463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790520929, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 14:55:29.329970 18547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:55:29.310994 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:56349 (local address 127.0.98.62:39531)
0927 14:55:29.311134 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0927 14:55:29.311139 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 14:55:29.312348 (+ 1209us) server_negotiation.cc:407] Connection header received
0927 14:55:29.313377 (+ 1029us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:55:29.313380 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:55:29.313434 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:55:29.313523 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:55:29.314779 (+ 1256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:55:29.315642 (+ 863us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:55:29.317339 (+ 1697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:55:29.317781 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:55:29.317904 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:55:29.326180 (+ 8276us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:55:29.326187 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:55:29.326191 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 14:55:29.326230 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:55:29.327755 (+ 1525us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:55:29.328269 (+ 514us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:55:29.328273 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:55:29.328275 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:55:29.328360 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:55:29.328727 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:55:29.328730 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:55:29.328732 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:55:29.328987 (+ 255us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:55:29.329117 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0927 14:55:29.329651 (+ 534us) server_negotiation.cc:299] Negotiation successful
0927 14:55:29.329793 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260927 14:55:29.330823 18701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:55:29.311453 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39531 (local address 127.0.98.1:56349)
0927 14:55:29.312122 (+ 669us) negotiation.cc:107] Waiting for socket to connect
0927 14:55:29.312175 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0927 14:55:29.313105 (+ 930us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:55:29.313677 (+ 572us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:55:29.313684 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:55:29.314140 (+ 456us) client_negotiation.cc:189] Negotiated authn=SASL
0927 14:55:29.314619 (+ 479us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:55:29.314631 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:55:29.315789 (+ 1158us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:55:29.315796 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:55:29.317227 (+ 1431us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:55:29.317236 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:55:29.317934 (+ 698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:55:29.317937 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:55:29.318074 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:55:29.323980 (+ 5906us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 14:55:29.323999 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0927 14:55:29.325928 (+ 1929us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 14:55:29.327865 (+ 1937us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 14:55:29.327871 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 14:55:29.327883 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0927 14:55:29.328146 (+ 263us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 14:55:29.328463 (+ 317us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 14:55:29.328466 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 14:55:29.328468 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 14:55:29.328619 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 14:55:29.329133 (+ 514us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 14:55:29.329141 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 14:55:29.329406 (+ 265us) client_negotiation.cc:769] Sending connection context
0927 14:55:29.329595 (+ 189us) client_negotiation.cc:240] Negotiation successful
0927 14:55:29.329938 (+ 343us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":477,"thread_start_us":116,"threads_started":1}
I20260927 14:55:29.332307 18699 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39531
I20260927 14:55:29.332585 18699 heartbeater.cc:464] Registering TS with master...
I20260927 14:55:29.333226 18699 heartbeater.cc:511] Master 127.0.98.62:39531 requested a full tablet report, sending...
I20260927 14:55:29.334810 18505 ts_manager.cc:194] Registered new tserver with Master: a9045ffd246740d1b1d8f42525689314 (127.0.98.1:37571)
I20260927 14:55:29.336162 18505 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:56349
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6936m5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790520684602251-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6936m5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790520684602251-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260927 14:55:29.342065 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6936m5/build/debug/bin/kudu
/tmp/dist-test-task6936m5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790520684602251-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790520684602251-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790520684602251-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790520684602251-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39531
--keytab_file=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790520684602251-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33327
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790520684602251-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790520684602251-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790520684602251-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 14:55:29.449038 18704 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 14:55:29.449317 18704 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 14:55:29.449384 18704 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 14:55:29.453331 18704 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 14:55:29.453403 18704 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 14:55:29.453490 18704 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 14:55:29.458452 18704 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33327
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790520684602251-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790520684602251-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790520684602251-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790520684602251-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39531
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18704
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790520684602251-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:15:17 UTC on ccycloud.quasar-ledxfe.root.comops.site
build id 4676134
I20260927 14:55:29.459378 18704 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 14:55:29.460299 18704 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 14:55:29.469353 18710 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 14:55:29.469411 18704 server_base.cc:1034] running on GCE node
W20260927 14:55:29.469429 18712 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 14:55:29.469362 18709 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 14:55:29.469908 18704 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 14:55:29.470484 18704 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 14:55:29.471665 18704 hybrid_clock.cc:648] HybridClock initialized: now 1790520929471655 us; error 26 us; skew 500 ppm
Sep 27 14:55:29 dist-test-slave-xk1g krb5kdc[18463](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790520929, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 14:55:29.474210 18704 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 14:55:29.474968 18704 webserver.cc:460] Webserver started at http://127.0.98.2:43065/ using document root <none> and password file <none>
I20260927 14:55:29.475492 18704 fs_manager.cc:359] Metadata directory not provided
I20260927 14:55:29.475579 18704 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 14:55:29.475795 18704 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 14:55:29.477828 18704 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6936m5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790520684602251-392-0/minicluster-data/ts-1/data/instance:
uuid: "fde5678a7e5d46fbbf51ea488750aa08"
format_stamp: "Formatted at 2026-09-27 14:55:29 on dist-test-slave-xk1g"
server_key: "ce81a9c75daf8290a0eda5c0d22c0409"
server_key_iv: "2d1d22decd043f574c02366e7f2f0be5"
server_key_version: "encryptionkey@0"
I20260927 14:55:29.478359 18704 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6936m5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790520684602251-392-0/minicluster-data/ts-1/wal/instance:
uuid: "fde5678a7e5d46fbbf51ea488750aa08"
format_stamp: "Formatted at 2026-09-27 14:55:29 on dist-test-slave-xk1g"
server_key: "ce81a9c75daf8290a0eda5c0d22c0409"
server_key_iv: "2d1d22decd043f574c02366e7f2f0be5"
server_key_version: "encryptionkey@0"
I20260927 14:55:29.481890 18704 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260927 14:55:29.484200 18719 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 14:55:29.485363 18704 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260927 14:55:29.485507 18704 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790520684602251-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task6936m5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790520684602251-392-0/minicluster-data/ts-1/wal
uuid: "fde5678a7e5d46fbbf51ea488750aa08"
format_stamp: "Formatted at 2026-09-27 14:55:29 on dist-test-slave-xk1g"
server_key: "ce81a9c75daf8290a0eda5c0d22c0409"
server_key_iv: "2d1d22decd043f574c02366e7f2f0be5"
server_key_version: "encryptionkey@0"
I20260927 14:55:29.485610 18704 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790520684602251-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790520684602251-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790520684602251-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 14:55:29.495530 18704 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 14:55:29.498139 18704 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 14:55:29.498318 18704 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 14:55:29.498826 18704 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 14:55:29.499557 18704 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 14:55:29.499631 18704 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 14:55:29.499715 18704 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 14:55:29.499766 18704 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 14:55:29.508790 18704 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:44357
I20260927 14:55:29.508809 18832 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:44357 every 8 connection(s)
I20260927 14:55:29.509738 18704 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6936m5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790520684602251-392-0/minicluster-data/ts-1/data/info.pb
I20260927 14:55:29.518301 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6936m5/build/debug/bin/kudu as pid 18704
I20260927 14:55:29.518445 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6936m5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790520684602251-392-0/minicluster-data/ts-1/wal/instance
I20260927 14:55:29.518697 392 external_mini_cluster.cc:1428] Setting key e4ab83ed7785a8ba8ac78feaf8062e23
Sep 27 14:55:29 dist-test-slave-xk1g krb5kdc[18463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790520929, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 14:55:29.524008 18547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:55:29.511482 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:45459 (local address 127.0.98.62:39531)
0927 14:55:29.511635 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0927 14:55:29.511647 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 14:55:29.512405 (+ 758us) server_negotiation.cc:407] Connection header received
0927 14:55:29.513205 (+ 800us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:55:29.513208 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:55:29.513253 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:55:29.513326 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:55:29.514411 (+ 1085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:55:29.515146 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:55:29.517071 (+ 1925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:55:29.517498 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:55:29.517611 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:55:29.520288 (+ 2677us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:55:29.520292 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:55:29.520294 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 14:55:29.520317 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:55:29.521877 (+ 1560us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:55:29.522377 (+ 500us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:55:29.522382 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:55:29.522384 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:55:29.522466 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:55:29.522848 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:55:29.522851 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:55:29.522852 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:55:29.523074 (+ 222us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:55:29.523156 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0927 14:55:29.523787 (+ 631us) server_negotiation.cc:299] Negotiation successful
0927 14:55:29.523890 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260927 14:55:29.525321 18835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:55:29.511845 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39531 (local address 127.0.98.2:45459)
0927 14:55:29.512260 (+ 415us) negotiation.cc:107] Waiting for socket to connect
0927 14:55:29.512300 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0927 14:55:29.513003 (+ 703us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:55:29.513475 (+ 472us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:55:29.513482 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:55:29.513810 (+ 328us) client_negotiation.cc:189] Negotiated authn=SASL
0927 14:55:29.514289 (+ 479us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:55:29.514302 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:55:29.515305 (+ 1003us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:55:29.515309 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:55:29.516920 (+ 1611us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:55:29.516934 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:55:29.517682 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:55:29.517687 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:55:29.517866 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:55:29.518502 (+ 636us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 14:55:29.518524 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0927 14:55:29.520110 (+ 1586us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 14:55:29.521990 (+ 1880us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 14:55:29.521996 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 14:55:29.522011 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0927 14:55:29.522271 (+ 260us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 14:55:29.522570 (+ 299us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 14:55:29.522572 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 14:55:29.522574 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 14:55:29.522724 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 14:55:29.523188 (+ 464us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 14:55:29.523197 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 14:55:29.523521 (+ 324us) client_negotiation.cc:769] Sending connection context
0927 14:55:29.523749 (+ 228us) client_negotiation.cc:240] Negotiation successful
0927 14:55:29.524131 (+ 382us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":239,"thread_start_us":104,"threads_started":1}
I20260927 14:55:29.526571 18833 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39531
I20260927 14:55:29.526809 18833 heartbeater.cc:464] Registering TS with master...
I20260927 14:55:29.527293 18833 heartbeater.cc:511] Master 127.0.98.62:39531 requested a full tablet report, sending...
I20260927 14:55:29.528347 18505 ts_manager.cc:194] Registered new tserver with Master: fde5678a7e5d46fbbf51ea488750aa08 (127.0.98.2:44357)
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260927 14:55:29.528934 18505 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:45459
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6936m5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790520684602251-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6936m5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790520684602251-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260927 14:55:29.545723 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6936m5/build/debug/bin/kudu
/tmp/dist-test-task6936m5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790520684602251-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790520684602251-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790520684602251-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790520684602251-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39531
--keytab_file=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790520684602251-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33327
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790520684602251-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790520684602251-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790520684602251-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 14:55:29.652698 18838 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 14:55:29.652978 18838 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 14:55:29.653074 18838 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 14:55:29.657024 18838 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 14:55:29.657137 18838 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 14:55:29.657226 18838 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 14:55:29.662402 18838 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33327
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790520684602251-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790520684602251-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790520684602251-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790520684602251-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39531
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18838
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790520684602251-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:15:17 UTC on ccycloud.quasar-ledxfe.root.comops.site
build id 4676134
I20260927 14:55:29.663411 18838 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 14:55:29.664418 18838 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 14:55:29.673851 18843 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 14:55:29.673871 18844 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 14:55:29.673939 18838 server_base.cc:1034] running on GCE node
W20260927 14:55:29.673874 18846 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 14:55:29.674557 18838 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 14:55:29.675128 18838 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 14:55:29.676326 18838 hybrid_clock.cc:648] HybridClock initialized: now 1790520929676304 us; error 33 us; skew 500 ppm
Sep 27 14:55:29 dist-test-slave-xk1g krb5kdc[18463](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790520929, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 14:55:29.678998 18838 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 14:55:29.679804 18838 webserver.cc:460] Webserver started at http://127.0.98.3:41117/ using document root <none> and password file <none>
I20260927 14:55:29.680300 18838 fs_manager.cc:359] Metadata directory not provided
I20260927 14:55:29.680356 18838 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 14:55:29.680510 18838 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 14:55:29.682446 18838 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6936m5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790520684602251-392-0/minicluster-data/ts-2/data/instance:
uuid: "6d6ad84cd46e41d09746b37a9fce5a70"
format_stamp: "Formatted at 2026-09-27 14:55:29 on dist-test-slave-xk1g"
server_key: "773cb652016f8ab080a39257a602f75d"
server_key_iv: "585a7208bd61a1ffdde6249359b794c8"
server_key_version: "encryptionkey@0"
I20260927 14:55:29.682933 18838 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6936m5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790520684602251-392-0/minicluster-data/ts-2/wal/instance:
uuid: "6d6ad84cd46e41d09746b37a9fce5a70"
format_stamp: "Formatted at 2026-09-27 14:55:29 on dist-test-slave-xk1g"
server_key: "773cb652016f8ab080a39257a602f75d"
server_key_iv: "585a7208bd61a1ffdde6249359b794c8"
server_key_version: "encryptionkey@0"
I20260927 14:55:29.686386 18838 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 14:55:29.688607 18853 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 14:55:29.689847 18838 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 14:55:29.689967 18838 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790520684602251-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task6936m5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790520684602251-392-0/minicluster-data/ts-2/wal
uuid: "6d6ad84cd46e41d09746b37a9fce5a70"
format_stamp: "Formatted at 2026-09-27 14:55:29 on dist-test-slave-xk1g"
server_key: "773cb652016f8ab080a39257a602f75d"
server_key_iv: "585a7208bd61a1ffdde6249359b794c8"
server_key_version: "encryptionkey@0"
I20260927 14:55:29.690080 18838 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790520684602251-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790520684602251-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790520684602251-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 14:55:29.702270 18838 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 14:55:29.704881 18838 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 14:55:29.705132 18838 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 14:55:29.705685 18838 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 14:55:29.706460 18838 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 14:55:29.706506 18838 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 14:55:29.706570 18838 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 14:55:29.706614 18838 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 14:55:29.715377 18838 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:39119
I20260927 14:55:29.715405 18966 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:39119 every 8 connection(s)
I20260927 14:55:29.716207 18838 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6936m5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790520684602251-392-0/minicluster-data/ts-2/data/info.pb
I20260927 14:55:29.722179 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6936m5/build/debug/bin/kudu as pid 18838
I20260927 14:55:29.722318 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6936m5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790520684602251-392-0/minicluster-data/ts-2/wal/instance
I20260927 14:55:29.722615 392 external_mini_cluster.cc:1428] Setting key 5d169c782b45a09aaa89b87d8c28dd77
Sep 27 14:55:29 dist-test-slave-xk1g krb5kdc[18463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790520929, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 14:55:29.728905 18547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:55:29.717861 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:49421 (local address 127.0.98.62:39531)
0927 14:55:29.717993 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0927 14:55:29.717996 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 14:55:29.718741 (+ 745us) server_negotiation.cc:407] Connection header received
0927 14:55:29.719524 (+ 783us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:55:29.719527 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:55:29.719583 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:55:29.719659 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:55:29.720764 (+ 1105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:55:29.721330 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:55:29.722849 (+ 1519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:55:29.723180 (+ 331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:55:29.723303 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:55:29.725899 (+ 2596us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:55:29.725901 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:55:29.725903 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 14:55:29.725925 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:55:29.727312 (+ 1387us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:55:29.727722 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:55:29.727724 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:55:29.727725 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:55:29.727794 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:55:29.728063 (+ 269us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:55:29.728065 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:55:29.728066 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:55:29.728228 (+ 162us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:55:29.728301 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0927 14:55:29.728706 (+ 405us) server_negotiation.cc:299] Negotiation successful
0927 14:55:29.728793 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260927 14:55:29.729686 18969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:55:29.718145 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39531 (local address 127.0.98.3:49421)
0927 14:55:29.718561 (+ 416us) negotiation.cc:107] Waiting for socket to connect
0927 14:55:29.718606 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0927 14:55:29.719340 (+ 734us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:55:29.719832 (+ 492us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:55:29.719839 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:55:29.720170 (+ 331us) client_negotiation.cc:189] Negotiated authn=SASL
0927 14:55:29.720619 (+ 449us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:55:29.720655 (+ 36us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:55:29.721455 (+ 800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:55:29.721458 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:55:29.722653 (+ 1195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:55:29.722664 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:55:29.723338 (+ 674us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:55:29.723341 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:55:29.723529 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:55:29.724115 (+ 586us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 14:55:29.724153 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0927 14:55:29.725758 (+ 1605us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 14:55:29.727402 (+ 1644us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 14:55:29.727406 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 14:55:29.727415 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0927 14:55:29.727642 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 14:55:29.727868 (+ 226us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 14:55:29.727870 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 14:55:29.727871 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 14:55:29.727990 (+ 119us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 14:55:29.728309 (+ 319us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 14:55:29.728314 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 14:55:29.728517 (+ 203us) client_negotiation.cc:769] Sending connection context
0927 14:55:29.728661 (+ 144us) client_negotiation.cc:240] Negotiation successful
0927 14:55:29.728896 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":248,"thread_start_us":106,"threads_started":1}
I20260927 14:55:29.730759 18967 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39531
I20260927 14:55:29.730976 18967 heartbeater.cc:464] Registering TS with master...
I20260927 14:55:29.731415 18967 heartbeater.cc:511] Master 127.0.98.62:39531 requested a full tablet report, sending...
I20260927 14:55:29.732347 18505 ts_manager.cc:194] Registered new tserver with Master: 6d6ad84cd46e41d09746b37a9fce5a70 (127.0.98.3:39119)
I20260927 14:55:29.732844 18505 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:49421
I20260927 14:55:29.736377 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 14:55:29.744805 18547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:55:29.738272 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49392 (local address 127.0.98.62:39531)
0927 14:55:29.738397 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0927 14:55:29.738400 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 14:55:29.738587 (+ 187us) server_negotiation.cc:407] Connection header received
0927 14:55:29.738677 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:55:29.738678 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:55:29.738709 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:55:29.738768 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:55:29.739365 (+ 597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:55:29.739827 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:55:29.740573 (+ 746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:55:29.740769 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:55:29.740838 (+ 69us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:55:29.742224 (+ 1386us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:55:29.742227 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:55:29.742229 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 14:55:29.742249 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:55:29.743392 (+ 1143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:55:29.743823 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:55:29.743826 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:55:29.743827 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:55:29.743890 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:55:29.744135 (+ 245us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:55:29.744137 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:55:29.744138 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:55:29.744320 (+ 182us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:55:29.744427 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0927 14:55:29.744639 (+ 212us) server_negotiation.cc:299] Negotiation successful
0927 14:55:29.744707 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260927 14:55:29.747360 18505 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49392:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 14:55:29.750387 18505 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 14:55:29.764114 18978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:55:29.757933 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:39119 (local address 127.0.0.1:58964)
0927 14:55:29.758272 (+ 339us) negotiation.cc:107] Waiting for socket to connect
0927 14:55:29.758317 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0927 14:55:29.758429 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:55:29.759738 (+ 1309us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:55:29.759746 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:55:29.759773 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 14:55:29.759996 (+ 223us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:55:29.760004 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:55:29.761363 (+ 1359us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:55:29.761369 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:55:29.762562 (+ 1193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:55:29.762570 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:55:29.763556 (+ 986us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:55:29.763560 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:55:29.763670 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:55:29.763711 (+ 41us) client_negotiation.cc:769] Sending connection context
0927 14:55:29.763830 (+ 119us) client_negotiation.cc:240] Negotiation successful
0927 14:55:29.763957 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":143,"mutex_wait_us":47,"thread_start_us":69,"threads_started":1}
I20260927 14:55:29.764228 18981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:55:29.758539 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58964 (local address 127.0.98.3:39119)
0927 14:55:29.759318 (+ 779us) server_negotiation.cc:206] Beginning negotiation
0927 14:55:29.759332 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 14:55:29.759355 (+ 23us) server_negotiation.cc:407] Connection header received
0927 14:55:29.759428 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:55:29.759432 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:55:29.759586 (+ 154us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:55:29.759705 (+ 119us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 14:55:29.760118 (+ 413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:55:29.761242 (+ 1124us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:55:29.762694 (+ 1452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:55:29.763429 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:55:29.763539 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:55:29.763692 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0927 14:55:29.763931 (+ 239us) server_negotiation.cc:299] Negotiation successful
0927 14:55:29.764054 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":662,"thread_start_us":104,"threads_started":1}
I20260927 14:55:29.764520 18979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:55:29.758354 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37571 (local address 127.0.0.1:52252)
0927 14:55:29.759033 (+ 679us) negotiation.cc:107] Waiting for socket to connect
0927 14:55:29.759046 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 14:55:29.759134 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:55:29.759813 (+ 679us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:55:29.759815 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:55:29.759836 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 14:55:29.759996 (+ 160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:55:29.760003 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:55:29.761613 (+ 1610us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:55:29.761616 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:55:29.762849 (+ 1233us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:55:29.762859 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:55:29.764098 (+ 1239us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:55:29.764102 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:55:29.764187 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:55:29.764210 (+ 23us) client_negotiation.cc:769] Sending connection context
0927 14:55:29.764282 (+ 72us) client_negotiation.cc:240] Negotiation successful
0927 14:55:29.764365 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":624,"mutex_wait_us":38,"thread_start_us":85,"threads_started":1}
I20260927 14:55:29.764925 18980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:55:29.758108 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:44357 (local address 127.0.0.1:34180)
0927 14:55:29.758585 (+ 477us) negotiation.cc:107] Waiting for socket to connect
0927 14:55:29.758598 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 14:55:29.758694 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:55:29.760486 (+ 1792us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:55:29.760489 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:55:29.760509 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 14:55:29.760585 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:55:29.760591 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:55:29.761947 (+ 1356us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:55:29.761950 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:55:29.763046 (+ 1096us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:55:29.763053 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:55:29.764131 (+ 1078us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:55:29.764137 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:55:29.764542 (+ 405us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:55:29.764655 (+ 113us) client_negotiation.cc:769] Sending connection context
0927 14:55:29.764737 (+ 82us) client_negotiation.cc:240] Negotiation successful
0927 14:55:29.764813 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":418,"thread_start_us":100,"threads_started":1}
I20260927 14:55:29.765239 18982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:55:29.758773 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52252 (local address 127.0.98.1:37571)
0927 14:55:29.759403 (+ 630us) server_negotiation.cc:206] Beginning negotiation
0927 14:55:29.759412 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 14:55:29.759430 (+ 18us) server_negotiation.cc:407] Connection header received
0927 14:55:29.759490 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:55:29.759493 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:55:29.759630 (+ 137us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:55:29.759821 (+ 191us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 14:55:29.760139 (+ 318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:55:29.761327 (+ 1188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:55:29.762977 (+ 1650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:55:29.763955 (+ 978us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:55:29.764797 (+ 842us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:55:29.764925 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0927 14:55:29.764972 (+ 47us) server_negotiation.cc:299] Negotiation successful
0927 14:55:29.765082 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":538,"thread_start_us":149,"threads_started":1}
I20260927 14:55:29.767676 18901 tablet_service.cc:1467] Processing CreateTablet for tablet 1f91b3cff3a74c2ab997813d6216bb32 (DEFAULT_TABLE table=test-table [id=2e5775af75db44bc9bc92fda49138a7e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 14:55:29.767973 18633 tablet_service.cc:1467] Processing CreateTablet for tablet 1f91b3cff3a74c2ab997813d6216bb32 (DEFAULT_TABLE table=test-table [id=2e5775af75db44bc9bc92fda49138a7e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 14:55:29.768615 18983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:55:29.759024 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34180 (local address 127.0.98.2:44357)
0927 14:55:29.760036 (+ 1012us) server_negotiation.cc:206] Beginning negotiation
0927 14:55:29.760049 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 14:55:29.760073 (+ 24us) server_negotiation.cc:407] Connection header received
0927 14:55:29.760148 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:55:29.760152 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:55:29.760337 (+ 185us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:55:29.760488 (+ 151us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 14:55:29.760724 (+ 236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:55:29.761809 (+ 1085us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:55:29.763152 (+ 1343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:55:29.763851 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:55:29.763941 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:55:29.764069 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0927 14:55:29.765201 (+ 1132us) server_negotiation.cc:299] Negotiation successful
0927 14:55:29.768471 (+ 3270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":910,"thread_start_us":90,"threads_started":1}
I20260927 14:55:29.768869 18901 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1f91b3cff3a74c2ab997813d6216bb32. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 14:55:29.768891 18633 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1f91b3cff3a74c2ab997813d6216bb32. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 14:55:29.771088 18767 tablet_service.cc:1467] Processing CreateTablet for tablet 1f91b3cff3a74c2ab997813d6216bb32 (DEFAULT_TABLE table=test-table [id=2e5775af75db44bc9bc92fda49138a7e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 14:55:29.771924 18767 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1f91b3cff3a74c2ab997813d6216bb32. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 14:55:29.776386 18984 tablet_bootstrap.cc:492] T 1f91b3cff3a74c2ab997813d6216bb32 P a9045ffd246740d1b1d8f42525689314: Bootstrap starting.
I20260927 14:55:29.777755 18985 tablet_bootstrap.cc:492] T 1f91b3cff3a74c2ab997813d6216bb32 P 6d6ad84cd46e41d09746b37a9fce5a70: Bootstrap starting.
I20260927 14:55:29.777827 18986 tablet_bootstrap.cc:492] T 1f91b3cff3a74c2ab997813d6216bb32 P fde5678a7e5d46fbbf51ea488750aa08: Bootstrap starting.
I20260927 14:55:29.778790 18984 tablet_bootstrap.cc:654] T 1f91b3cff3a74c2ab997813d6216bb32 P a9045ffd246740d1b1d8f42525689314: Neither blocks nor log segments found. Creating new log.
I20260927 14:55:29.779810 18984 log.cc:824] T 1f91b3cff3a74c2ab997813d6216bb32 P a9045ffd246740d1b1d8f42525689314: Log is configured to *not* fsync() on all Append() calls
I20260927 14:55:29.779969 18985 tablet_bootstrap.cc:654] T 1f91b3cff3a74c2ab997813d6216bb32 P 6d6ad84cd46e41d09746b37a9fce5a70: Neither blocks nor log segments found. Creating new log.
I20260927 14:55:29.780373 18986 tablet_bootstrap.cc:654] T 1f91b3cff3a74c2ab997813d6216bb32 P fde5678a7e5d46fbbf51ea488750aa08: Neither blocks nor log segments found. Creating new log.
I20260927 14:55:29.781275 18986 log.cc:824] T 1f91b3cff3a74c2ab997813d6216bb32 P fde5678a7e5d46fbbf51ea488750aa08: Log is configured to *not* fsync() on all Append() calls
I20260927 14:55:29.781275 18985 log.cc:824] T 1f91b3cff3a74c2ab997813d6216bb32 P 6d6ad84cd46e41d09746b37a9fce5a70: Log is configured to *not* fsync() on all Append() calls
I20260927 14:55:29.782136 18984 tablet_bootstrap.cc:492] T 1f91b3cff3a74c2ab997813d6216bb32 P a9045ffd246740d1b1d8f42525689314: No bootstrap required, opened a new log
I20260927 14:55:29.782337 18984 ts_tablet_manager.cc:1397] T 1f91b3cff3a74c2ab997813d6216bb32 P a9045ffd246740d1b1d8f42525689314: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260927 14:55:29.783285 18985 tablet_bootstrap.cc:492] T 1f91b3cff3a74c2ab997813d6216bb32 P 6d6ad84cd46e41d09746b37a9fce5a70: No bootstrap required, opened a new log
I20260927 14:55:29.783473 18986 tablet_bootstrap.cc:492] T 1f91b3cff3a74c2ab997813d6216bb32 P fde5678a7e5d46fbbf51ea488750aa08: No bootstrap required, opened a new log
I20260927 14:55:29.783468 18985 ts_tablet_manager.cc:1397] T 1f91b3cff3a74c2ab997813d6216bb32 P 6d6ad84cd46e41d09746b37a9fce5a70: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.004s
I20260927 14:55:29.783649 18986 ts_tablet_manager.cc:1397] T 1f91b3cff3a74c2ab997813d6216bb32 P fde5678a7e5d46fbbf51ea488750aa08: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260927 14:55:29.785336 18984 raft_consensus.cc:348] T 1f91b3cff3a74c2ab997813d6216bb32 P a9045ffd246740d1b1d8f42525689314 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d6ad84cd46e41d09746b37a9fce5a70" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39119 } } peers { permanent_uuid: "fde5678a7e5d46fbbf51ea488750aa08" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44357 } } peers { permanent_uuid: "a9045ffd246740d1b1d8f42525689314" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37571 } }
I20260927 14:55:29.785565 18984 raft_consensus.cc:374] T 1f91b3cff3a74c2ab997813d6216bb32 P a9045ffd246740d1b1d8f42525689314 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 14:55:29.785616 18984 raft_consensus.cc:729] T 1f91b3cff3a74c2ab997813d6216bb32 P a9045ffd246740d1b1d8f42525689314 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a9045ffd246740d1b1d8f42525689314, State: Initialized, Role: FOLLOWER
I20260927 14:55:29.786033 18984 consensus_queue.cc:260] T 1f91b3cff3a74c2ab997813d6216bb32 P a9045ffd246740d1b1d8f42525689314 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d6ad84cd46e41d09746b37a9fce5a70" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39119 } } peers { permanent_uuid: "fde5678a7e5d46fbbf51ea488750aa08" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44357 } } peers { permanent_uuid: "a9045ffd246740d1b1d8f42525689314" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37571 } }
I20260927 14:55:29.786635 18699 heartbeater.cc:503] Master 127.0.98.62:39531 was elected leader, sending a full tablet report...
I20260927 14:55:29.787127 18984 ts_tablet_manager.cc:1428] T 1f91b3cff3a74c2ab997813d6216bb32 P a9045ffd246740d1b1d8f42525689314: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260927 14:55:29.787496 18985 raft_consensus.cc:348] T 1f91b3cff3a74c2ab997813d6216bb32 P 6d6ad84cd46e41d09746b37a9fce5a70 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d6ad84cd46e41d09746b37a9fce5a70" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39119 } } peers { permanent_uuid: "fde5678a7e5d46fbbf51ea488750aa08" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44357 } } peers { permanent_uuid: "a9045ffd246740d1b1d8f42525689314" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37571 } }
I20260927 14:55:29.787496 18986 raft_consensus.cc:348] T 1f91b3cff3a74c2ab997813d6216bb32 P fde5678a7e5d46fbbf51ea488750aa08 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d6ad84cd46e41d09746b37a9fce5a70" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39119 } } peers { permanent_uuid: "fde5678a7e5d46fbbf51ea488750aa08" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44357 } } peers { permanent_uuid: "a9045ffd246740d1b1d8f42525689314" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37571 } }
I20260927 14:55:29.787774 18985 raft_consensus.cc:374] T 1f91b3cff3a74c2ab997813d6216bb32 P 6d6ad84cd46e41d09746b37a9fce5a70 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 14:55:29.787774 18986 raft_consensus.cc:374] T 1f91b3cff3a74c2ab997813d6216bb32 P fde5678a7e5d46fbbf51ea488750aa08 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 14:55:29.787828 18986 raft_consensus.cc:729] T 1f91b3cff3a74c2ab997813d6216bb32 P fde5678a7e5d46fbbf51ea488750aa08 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fde5678a7e5d46fbbf51ea488750aa08, State: Initialized, Role: FOLLOWER
I20260927 14:55:29.787828 18985 raft_consensus.cc:729] T 1f91b3cff3a74c2ab997813d6216bb32 P 6d6ad84cd46e41d09746b37a9fce5a70 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6d6ad84cd46e41d09746b37a9fce5a70, State: Initialized, Role: FOLLOWER
I20260927 14:55:29.788269 18985 consensus_queue.cc:260] T 1f91b3cff3a74c2ab997813d6216bb32 P 6d6ad84cd46e41d09746b37a9fce5a70 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d6ad84cd46e41d09746b37a9fce5a70" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39119 } } peers { permanent_uuid: "fde5678a7e5d46fbbf51ea488750aa08" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44357 } } peers { permanent_uuid: "a9045ffd246740d1b1d8f42525689314" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37571 } }
I20260927 14:55:29.788269 18986 consensus_queue.cc:260] T 1f91b3cff3a74c2ab997813d6216bb32 P fde5678a7e5d46fbbf51ea488750aa08 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d6ad84cd46e41d09746b37a9fce5a70" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39119 } } peers { permanent_uuid: "fde5678a7e5d46fbbf51ea488750aa08" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44357 } } peers { permanent_uuid: "a9045ffd246740d1b1d8f42525689314" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37571 } }
I20260927 14:55:29.788967 18833 heartbeater.cc:503] Master 127.0.98.62:39531 was elected leader, sending a full tablet report...
I20260927 14:55:29.789202 18967 heartbeater.cc:503] Master 127.0.98.62:39531 was elected leader, sending a full tablet report...
I20260927 14:55:29.790560 18986 ts_tablet_manager.cc:1428] T 1f91b3cff3a74c2ab997813d6216bb32 P fde5678a7e5d46fbbf51ea488750aa08: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260927 14:55:29.790580 18985 ts_tablet_manager.cc:1428] T 1f91b3cff3a74c2ab997813d6216bb32 P 6d6ad84cd46e41d09746b37a9fce5a70: Time spent starting tablet: real 0.007s user 0.006s sys 0.001s
W20260927 14:55:29.810370 18700 tablet.cc:2369] T 1f91b3cff3a74c2ab997813d6216bb32 P a9045ffd246740d1b1d8f42525689314: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 14:55:29.967124 18968 tablet.cc:2369] T 1f91b3cff3a74c2ab997813d6216bb32 P 6d6ad84cd46e41d09746b37a9fce5a70: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 14:55:29.969870 18991 raft_consensus.cc:482] T 1f91b3cff3a74c2ab997813d6216bb32 P fde5678a7e5d46fbbf51ea488750aa08 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 14:55:29.970084 18991 raft_consensus.cc:504] T 1f91b3cff3a74c2ab997813d6216bb32 P fde5678a7e5d46fbbf51ea488750aa08 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d6ad84cd46e41d09746b37a9fce5a70" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39119 } } peers { permanent_uuid: "fde5678a7e5d46fbbf51ea488750aa08" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44357 } } peers { permanent_uuid: "a9045ffd246740d1b1d8f42525689314" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37571 } }
I20260927 14:55:29.971321 18991 leader_election.cc:288] T 1f91b3cff3a74c2ab997813d6216bb32 P fde5678a7e5d46fbbf51ea488750aa08 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6d6ad84cd46e41d09746b37a9fce5a70 (127.0.98.3:39119), a9045ffd246740d1b1d8f42525689314 (127.0.98.1:37571)
I20260927 14:55:29.975092 18981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:55:29.971769 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:53089 (local address 127.0.98.3:39119)
0927 14:55:29.971901 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0927 14:55:29.971905 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 14:55:29.971917 (+ 12us) server_negotiation.cc:407] Connection header received
0927 14:55:29.971962 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:55:29.971964 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:55:29.972002 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:55:29.972105 (+ 103us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 14:55:29.972361 (+ 256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:55:29.972861 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:55:29.974066 (+ 1205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:55:29.974476 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:55:29.974612 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:55:29.974674 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0927 14:55:29.974893 (+ 219us) server_negotiation.cc:299] Negotiation successful
0927 14:55:29.974968 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260927 14:55:29.975092 18835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:55:29.971543 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:39119 (local address 127.0.98.2:53089)
0927 14:55:29.971666 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0927 14:55:29.971686 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0927 14:55:29.971785 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:55:29.972117 (+ 332us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:55:29.972119 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:55:29.972137 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 14:55:29.972253 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:55:29.972258 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:55:29.972973 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:55:29.972978 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:55:29.973927 (+ 949us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:55:29.973936 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:55:29.974618 (+ 682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:55:29.974622 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:55:29.974726 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:55:29.974786 (+ 60us) client_negotiation.cc:769] Sending connection context
0927 14:55:29.974873 (+ 87us) client_negotiation.cc:240] Negotiation successful
0927 14:55:29.974954 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":37,"mutex_wait_us":27}
I20260927 14:55:29.975198 18982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:55:29.971685 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:40599 (local address 127.0.98.1:37571)
0927 14:55:29.971809 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0927 14:55:29.971813 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 14:55:29.972085 (+ 272us) server_negotiation.cc:407] Connection header received
0927 14:55:29.972246 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:55:29.972250 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:55:29.972297 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:55:29.972386 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 14:55:29.972636 (+ 250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:55:29.973144 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:55:29.974184 (+ 1040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:55:29.974602 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:55:29.974722 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:55:29.974799 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0927 14:55:29.974990 (+ 191us) server_negotiation.cc:299] Negotiation successful
0927 14:55:29.975066 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260927 14:55:29.975454 18993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:55:29.971595 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37571 (local address 127.0.98.2:40599)
0927 14:55:29.971987 (+ 392us) negotiation.cc:107] Waiting for socket to connect
0927 14:55:29.971998 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 14:55:29.972125 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:55:29.972398 (+ 273us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:55:29.972401 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:55:29.972416 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 14:55:29.972528 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:55:29.972533 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:55:29.973278 (+ 745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:55:29.973285 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:55:29.974054 (+ 769us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:55:29.974062 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:55:29.974727 (+ 665us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:55:29.974730 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:55:29.974869 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:55:29.974887 (+ 18us) client_negotiation.cc:769] Sending connection context
0927 14:55:29.974973 (+ 86us) client_negotiation.cc:240] Negotiation successful
0927 14:55:29.975040 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":343,"mutex_wait_us":3,"thread_start_us":133,"threads_started":1}
I20260927 14:55:29.975852 18921 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f91b3cff3a74c2ab997813d6216bb32" candidate_uuid: "fde5678a7e5d46fbbf51ea488750aa08" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6d6ad84cd46e41d09746b37a9fce5a70" is_pre_election: true
I20260927 14:55:29.976178 18921 raft_consensus.cc:2454] T 1f91b3cff3a74c2ab997813d6216bb32 P 6d6ad84cd46e41d09746b37a9fce5a70 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fde5678a7e5d46fbbf51ea488750aa08 in term 0.
I20260927 14:55:29.976279 18653 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f91b3cff3a74c2ab997813d6216bb32" candidate_uuid: "fde5678a7e5d46fbbf51ea488750aa08" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a9045ffd246740d1b1d8f42525689314" is_pre_election: true
I20260927 14:55:29.976583 18653 raft_consensus.cc:2454] T 1f91b3cff3a74c2ab997813d6216bb32 P a9045ffd246740d1b1d8f42525689314 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fde5678a7e5d46fbbf51ea488750aa08 in term 0.
I20260927 14:55:29.976651 18721 leader_election.cc:302] T 1f91b3cff3a74c2ab997813d6216bb32 P fde5678a7e5d46fbbf51ea488750aa08 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d6ad84cd46e41d09746b37a9fce5a70, fde5678a7e5d46fbbf51ea488750aa08; no voters:
I20260927 14:55:29.977012 18991 raft_consensus.cc:2781] T 1f91b3cff3a74c2ab997813d6216bb32 P fde5678a7e5d46fbbf51ea488750aa08 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 14:55:29.977135 18991 raft_consensus.cc:482] T 1f91b3cff3a74c2ab997813d6216bb32 P fde5678a7e5d46fbbf51ea488750aa08 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 14:55:29.977200 18991 raft_consensus.cc:3037] T 1f91b3cff3a74c2ab997813d6216bb32 P fde5678a7e5d46fbbf51ea488750aa08 [term 0 FOLLOWER]: Advancing to term 1
I20260927 14:55:29.978201 18991 raft_consensus.cc:504] T 1f91b3cff3a74c2ab997813d6216bb32 P fde5678a7e5d46fbbf51ea488750aa08 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d6ad84cd46e41d09746b37a9fce5a70" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39119 } } peers { permanent_uuid: "fde5678a7e5d46fbbf51ea488750aa08" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44357 } } peers { permanent_uuid: "a9045ffd246740d1b1d8f42525689314" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37571 } }
I20260927 14:55:29.978545 18991 leader_election.cc:288] T 1f91b3cff3a74c2ab997813d6216bb32 P fde5678a7e5d46fbbf51ea488750aa08 [CANDIDATE]: Term 1 election: Requested vote from peers 6d6ad84cd46e41d09746b37a9fce5a70 (127.0.98.3:39119), a9045ffd246740d1b1d8f42525689314 (127.0.98.1:37571)
I20260927 14:55:29.978915 18653 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f91b3cff3a74c2ab997813d6216bb32" candidate_uuid: "fde5678a7e5d46fbbf51ea488750aa08" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a9045ffd246740d1b1d8f42525689314"
I20260927 14:55:29.978906 18921 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f91b3cff3a74c2ab997813d6216bb32" candidate_uuid: "fde5678a7e5d46fbbf51ea488750aa08" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6d6ad84cd46e41d09746b37a9fce5a70"
I20260927 14:55:29.979028 18653 raft_consensus.cc:3037] T 1f91b3cff3a74c2ab997813d6216bb32 P a9045ffd246740d1b1d8f42525689314 [term 0 FOLLOWER]: Advancing to term 1
I20260927 14:55:29.979028 18921 raft_consensus.cc:3037] T 1f91b3cff3a74c2ab997813d6216bb32 P 6d6ad84cd46e41d09746b37a9fce5a70 [term 0 FOLLOWER]: Advancing to term 1
I20260927 14:55:29.980276 18921 raft_consensus.cc:2454] T 1f91b3cff3a74c2ab997813d6216bb32 P 6d6ad84cd46e41d09746b37a9fce5a70 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fde5678a7e5d46fbbf51ea488750aa08 in term 1.
I20260927 14:55:29.980275 18653 raft_consensus.cc:2454] T 1f91b3cff3a74c2ab997813d6216bb32 P a9045ffd246740d1b1d8f42525689314 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fde5678a7e5d46fbbf51ea488750aa08 in term 1.
I20260927 14:55:29.980561 18721 leader_election.cc:302] T 1f91b3cff3a74c2ab997813d6216bb32 P fde5678a7e5d46fbbf51ea488750aa08 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d6ad84cd46e41d09746b37a9fce5a70, fde5678a7e5d46fbbf51ea488750aa08; no voters:
I20260927 14:55:29.980723 18991 raft_consensus.cc:2781] T 1f91b3cff3a74c2ab997813d6216bb32 P fde5678a7e5d46fbbf51ea488750aa08 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 14:55:29.980995 18991 raft_consensus.cc:686] T 1f91b3cff3a74c2ab997813d6216bb32 P fde5678a7e5d46fbbf51ea488750aa08 [term 1 LEADER]: Becoming Leader. State: Replica: fde5678a7e5d46fbbf51ea488750aa08, State: Running, Role: LEADER
I20260927 14:55:29.981369 18991 consensus_queue.cc:237] T 1f91b3cff3a74c2ab997813d6216bb32 P fde5678a7e5d46fbbf51ea488750aa08 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d6ad84cd46e41d09746b37a9fce5a70" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39119 } } peers { permanent_uuid: "fde5678a7e5d46fbbf51ea488750aa08" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44357 } } peers { permanent_uuid: "a9045ffd246740d1b1d8f42525689314" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37571 } }
I20260927 14:55:29.984506 18508 catalog_manager.cc:5495] T 1f91b3cff3a74c2ab997813d6216bb32 P fde5678a7e5d46fbbf51ea488750aa08 reported cstate change: term changed from 0 to 1, leader changed from <none> to fde5678a7e5d46fbbf51ea488750aa08 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "fde5678a7e5d46fbbf51ea488750aa08" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d6ad84cd46e41d09746b37a9fce5a70" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39119 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fde5678a7e5d46fbbf51ea488750aa08" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44357 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a9045ffd246740d1b1d8f42525689314" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37571 } health_report { overall_health: UNKNOWN } } }
I20260927 14:55:30.000934 18983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:55:29.997778 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34190 (local address 127.0.98.2:44357)
0927 14:55:29.997900 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0927 14:55:29.997904 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 14:55:29.997917 (+ 13us) server_negotiation.cc:407] Connection header received
0927 14:55:29.997999 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:55:29.998001 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:55:29.998054 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:55:29.998145 (+ 91us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 14:55:29.998351 (+ 206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:55:29.998850 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:55:29.999625 (+ 775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:55:29.999835 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:55:29.999908 (+ 73us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:55:30.000173 (+ 265us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 14:55:30.000573 (+ 400us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 14:55:30.000697 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0927 14:55:30.000779 (+ 82us) server_negotiation.cc:299] Negotiation successful
0927 14:55:30.000810 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260927 14:55:30.006906 18653 raft_consensus.cc:1261] T 1f91b3cff3a74c2ab997813d6216bb32 P a9045ffd246740d1b1d8f42525689314 [term 1 FOLLOWER]: Refusing update from remote peer fde5678a7e5d46fbbf51ea488750aa08: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 14:55:30.007005 18921 raft_consensus.cc:1261] T 1f91b3cff3a74c2ab997813d6216bb32 P 6d6ad84cd46e41d09746b37a9fce5a70 [term 1 FOLLOWER]: Refusing update from remote peer fde5678a7e5d46fbbf51ea488750aa08: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 14:55:30.007704 18994 consensus_queue.cc:1035] T 1f91b3cff3a74c2ab997813d6216bb32 P fde5678a7e5d46fbbf51ea488750aa08 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a9045ffd246740d1b1d8f42525689314" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37571 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 14:55:30.007881 18991 consensus_queue.cc:1035] T 1f91b3cff3a74c2ab997813d6216bb32 P fde5678a7e5d46fbbf51ea488750aa08 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d6ad84cd46e41d09746b37a9fce5a70" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260927 14:55:30.010767 18834 tablet.cc:2369] T 1f91b3cff3a74c2ab997813d6216bb32 P fde5678a7e5d46fbbf51ea488750aa08: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 14:55:30.023926 18502 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49392:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 14:55:30.024112 18502 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49392:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 14:55:30.026703 18502 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5184c6a24d254960ae0f5ff3006e82af: Sending DeleteTablet for 3 replicas of tablet 1f91b3cff3a74c2ab997813d6216bb32
I20260927 14:55:30.027410 18901 tablet_service.cc:1514] Processing DeleteTablet for tablet 1f91b3cff3a74c2ab997813d6216bb32 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 14:55:30 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:58964
I20260927 14:55:30.027593 18633 tablet_service.cc:1514] Processing DeleteTablet for tablet 1f91b3cff3a74c2ab997813d6216bb32 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 14:55:30 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:52252
I20260927 14:55:30.028049 19008 tablet_replica.cc:330] T 1f91b3cff3a74c2ab997813d6216bb32 P a9045ffd246740d1b1d8f42525689314: stopping tablet replica
I20260927 14:55:30.028362 19008 raft_consensus.cc:2229] T 1f91b3cff3a74c2ab997813d6216bb32 P a9045ffd246740d1b1d8f42525689314 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 14:55:30.028654 19008 raft_consensus.cc:2258] T 1f91b3cff3a74c2ab997813d6216bb32 P a9045ffd246740d1b1d8f42525689314 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 14:55:30.028779 18767 tablet_service.cc:1514] Processing DeleteTablet for tablet 1f91b3cff3a74c2ab997813d6216bb32 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 14:55:30 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:34180
I20260927 14:55:30.029300 19009 tablet_replica.cc:330] T 1f91b3cff3a74c2ab997813d6216bb32 P fde5678a7e5d46fbbf51ea488750aa08: stopping tablet replica
I20260927 14:55:30.029472 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6936m5/build/debug/bin/kudu with pid 18570
I20260927 14:55:30.029562 19009 raft_consensus.cc:2229] T 1f91b3cff3a74c2ab997813d6216bb32 P fde5678a7e5d46fbbf51ea488750aa08 [term 1 LEADER]: Raft consensus shutting down.
I20260927 14:55:30.029836 19009 raft_consensus.cc:2258] T 1f91b3cff3a74c2ab997813d6216bb32 P fde5678a7e5d46fbbf51ea488750aa08 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 14:55:30.031950 19009 ts_tablet_manager.cc:1905] T 1f91b3cff3a74c2ab997813d6216bb32 P fde5678a7e5d46fbbf51ea488750aa08: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 14:55:30.034713 19009 ts_tablet_manager.cc:1918] T 1f91b3cff3a74c2ab997813d6216bb32 P fde5678a7e5d46fbbf51ea488750aa08: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 14:55:30.034803 19009 log.cc:1196] T 1f91b3cff3a74c2ab997813d6216bb32 P fde5678a7e5d46fbbf51ea488750aa08: Deleting WAL directory at /tmp/dist-test-task6936m5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790520684602251-392-0/minicluster-data/ts-1/wal/wals/1f91b3cff3a74c2ab997813d6216bb32
I20260927 14:55:30.035094 19009 ts_tablet_manager.cc:1939] T 1f91b3cff3a74c2ab997813d6216bb32 P fde5678a7e5d46fbbf51ea488750aa08: Deleting consensus metadata
I20260927 14:55:30.036271 18495 catalog_manager.cc:4841] TS fde5678a7e5d46fbbf51ea488750aa08 (127.0.98.2:44357): tablet 1f91b3cff3a74c2ab997813d6216bb32 (table test-table [id=2e5775af75db44bc9bc92fda49138a7e]) successfully deleted
I20260927 14:55:30.037294 19007 tablet_replica.cc:330] T 1f91b3cff3a74c2ab997813d6216bb32 P 6d6ad84cd46e41d09746b37a9fce5a70: stopping tablet replica
I20260927 14:55:30.037668 19007 raft_consensus.cc:2229] T 1f91b3cff3a74c2ab997813d6216bb32 P 6d6ad84cd46e41d09746b37a9fce5a70 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 14:55:30.038025 19007 raft_consensus.cc:2258] T 1f91b3cff3a74c2ab997813d6216bb32 P 6d6ad84cd46e41d09746b37a9fce5a70 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260927 14:55:30.039059 18494 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:37571 (error 108)
I20260927 14:55:30.039745 19007 ts_tablet_manager.cc:1905] T 1f91b3cff3a74c2ab997813d6216bb32 P 6d6ad84cd46e41d09746b37a9fce5a70: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 14:55:30.039809 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6936m5/build/debug/bin/kudu with pid 18704
I20260927 14:55:30.042649 19007 ts_tablet_manager.cc:1918] T 1f91b3cff3a74c2ab997813d6216bb32 P 6d6ad84cd46e41d09746b37a9fce5a70: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 14:55:30.042804 18980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:55:30.042452 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37571 (local address 127.0.0.1:52258)
0927 14:55:30.042611 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0927 14:55:30.042696 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:37571: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20260927 14:55:30.042807 19007 log.cc:1196] T 1f91b3cff3a74c2ab997813d6216bb32 P 6d6ad84cd46e41d09746b37a9fce5a70: Deleting WAL directory at /tmp/dist-test-task6936m5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790520684602251-392-0/minicluster-data/ts-2/wal/wals/1f91b3cff3a74c2ab997813d6216bb32
W20260927 14:55:30.042951 18494 catalog_manager.cc:4584] TS a9045ffd246740d1b1d8f42525689314 (127.0.98.1:37571): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 1f91b3cff3a74c2ab997813d6216bb32: Network error: Client connection negotiation failed: client connection to 127.0.98.1:37571: connect: Connection refused (error 111)
I20260927 14:55:30.043322 19007 ts_tablet_manager.cc:1939] T 1f91b3cff3a74c2ab997813d6216bb32 P 6d6ad84cd46e41d09746b37a9fce5a70: Deleting consensus metadata
I20260927 14:55:30.044477 18493 catalog_manager.cc:4841] TS 6d6ad84cd46e41d09746b37a9fce5a70 (127.0.98.3:39119): tablet 1f91b3cff3a74c2ab997813d6216bb32 (table test-table [id=2e5775af75db44bc9bc92fda49138a7e]) successfully deleted
I20260927 14:55:30.048448 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6936m5/build/debug/bin/kudu with pid 18838
I20260927 14:55:30.054607 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6936m5/build/debug/bin/kudu with pid 18476
2026-09-27T14:55:30Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (2355 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-09-27T14:55:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T14:55:30Z Disabled control of system clock
I20260927 14:55:30.084200 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6936m5/build/debug/bin/kudu
/tmp/dist-test-task6936m5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790520684602251-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790520684602251-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790520684602251-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790520684602251-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37253
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:41419
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:37253
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260927 14:55:30.189730 19014 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 14:55:30.189966 19014 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 14:55:30.190001 19014 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 14:55:30.193871 19014 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 14:55:30.193917 19014 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 14:55:30.193933 19014 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 14:55:30.193949 19014 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 14:55:30.193964 19014 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 14:55:30.198487 19014 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41419
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790520684602251-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790520684602251-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:37253
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37253
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790520684602251-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19014
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790520684602251-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:15:17 UTC on ccycloud.quasar-ledxfe.root.comops.site
build id 4676134
I20260927 14:55:30.199410 19014 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 14:55:30.200641 19014 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 14:55:30.208614 19020 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 14:55:30.208608 19022 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 14:55:30.208654 19014 server_base.cc:1034] running on GCE node
W20260927 14:55:30.208607 19019 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 14:55:30.209450 19014 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 14:55:30.210461 19014 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 14:55:30.211614 19014 hybrid_clock.cc:648] HybridClock initialized: now 1790520930211586 us; error 40 us; skew 500 ppm
I20260927 14:55:30.213558 19014 webserver.cc:460] Webserver started at http://127.0.98.62:46671/ using document root <none> and password file <none>
I20260927 14:55:30.214111 19014 fs_manager.cc:359] Metadata directory not provided
I20260927 14:55:30.214171 19014 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 14:55:30.214416 19014 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 14:55:30.216360 19014 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6936m5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790520684602251-392-0/minicluster-data/master-0/data/instance:
uuid: "5917088264224f6ba2219c70f2512fe3"
format_stamp: "Formatted at 2026-09-27 14:55:30 on dist-test-slave-xk1g"
server_key: "0f5418220cdf75820d535c67e0612500"
server_key_iv: "882197b4fe91c869b775af0b015e3f78"
server_key_version: "encryptionkey@0"
I20260927 14:55:30.216893 19014 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6936m5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790520684602251-392-0/minicluster-data/master-0/wal/instance:
uuid: "5917088264224f6ba2219c70f2512fe3"
format_stamp: "Formatted at 2026-09-27 14:55:30 on dist-test-slave-xk1g"
server_key: "0f5418220cdf75820d535c67e0612500"
server_key_iv: "882197b4fe91c869b775af0b015e3f78"
server_key_version: "encryptionkey@0"
I20260927 14:55:30.220287 19014 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260927 14:55:30.222556 19028 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 14:55:30.223673 19014 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 14:55:30.223820 19014 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790520684602251-392-0/minicluster-data/master-0/data,/tmp/dist-test-task6936m5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790520684602251-392-0/minicluster-data/master-0/wal
uuid: "5917088264224f6ba2219c70f2512fe3"
format_stamp: "Formatted at 2026-09-27 14:55:30 on dist-test-slave-xk1g"
server_key: "0f5418220cdf75820d535c67e0612500"
server_key_iv: "882197b4fe91c869b775af0b015e3f78"
server_key_version: "encryptionkey@0"
I20260927 14:55:30.223918 19014 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790520684602251-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790520684602251-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790520684602251-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 14:55:30.252239 19014 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 14:55:30.252969 19014 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 14:55:30.253192 19014 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 14:55:30.260080 19014 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:37253
I20260927 14:55:30.260109 19080 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:37253 every 8 connection(s)
I20260927 14:55:30.261019 19014 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6936m5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790520684602251-392-0/minicluster-data/master-0/data/info.pb
I20260927 14:55:30.264020 19081 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 14:55:30.269407 19081 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5917088264224f6ba2219c70f2512fe3: Bootstrap starting.
I20260927 14:55:30.269482 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6936m5/build/debug/bin/kudu as pid 19014
I20260927 14:55:30.269604 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6936m5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790520684602251-392-0/minicluster-data/master-0/wal/instance
I20260927 14:55:30.269876 392 external_mini_cluster.cc:1428] Setting key 257e320826f55fa82779764dca4b0f2a
I20260927 14:55:30.272068 19081 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5917088264224f6ba2219c70f2512fe3: Neither blocks nor log segments found. Creating new log.
I20260927 14:55:30.272773 19081 log.cc:824] T 00000000000000000000000000000000 P 5917088264224f6ba2219c70f2512fe3: Log is configured to *not* fsync() on all Append() calls
I20260927 14:55:30.274627 19081 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5917088264224f6ba2219c70f2512fe3: No bootstrap required, opened a new log
I20260927 14:55:30.278232 19081 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5917088264224f6ba2219c70f2512fe3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5917088264224f6ba2219c70f2512fe3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37253 } }
I20260927 14:55:30.278430 19081 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5917088264224f6ba2219c70f2512fe3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 14:55:30.278488 19081 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5917088264224f6ba2219c70f2512fe3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5917088264224f6ba2219c70f2512fe3, State: Initialized, Role: FOLLOWER
I20260927 14:55:30.278842 19084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:55:30.271227 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44580 (local address 127.0.98.62:37253)
0927 14:55:30.271849 (+ 622us) server_negotiation.cc:206] Beginning negotiation
0927 14:55:30.271864 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 14:55:30.271897 (+ 33us) server_negotiation.cc:407] Connection header received
0927 14:55:30.272549 (+ 652us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:55:30.272568 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:55:30.272995 (+ 427us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:55:30.273383 (+ 388us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:55:30.273738 (+ 355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:55:30.274704 (+ 966us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:55:30.275575 (+ 871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:55:30.276093 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:55:30.276271 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:55:30.277119 (+ 848us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:55:30.277124 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:55:30.277136 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 14:55:30.277501 (+ 365us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:55:30.277552 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 14:55:30.277570 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:55:30.277672 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0927 14:55:30.277842 (+ 170us) server_negotiation.cc:299] Negotiation successful
0927 14:55:30.278081 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":265,"thread_start_us":119,"threads_started":1}
I20260927 14:55:30.278915 19081 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5917088264224f6ba2219c70f2512fe3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5917088264224f6ba2219c70f2512fe3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37253 } }
I20260927 14:55:30.279078 19081 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5917088264224f6ba2219c70f2512fe3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 14:55:30.279147 19081 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5917088264224f6ba2219c70f2512fe3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 14:55:30.279248 19081 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5917088264224f6ba2219c70f2512fe3 [term 0 FOLLOWER]: Advancing to term 1
I20260927 14:55:30.280164 19081 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5917088264224f6ba2219c70f2512fe3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5917088264224f6ba2219c70f2512fe3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37253 } }
I20260927 14:55:30.280483 19081 leader_election.cc:302] T 00000000000000000000000000000000 P 5917088264224f6ba2219c70f2512fe3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5917088264224f6ba2219c70f2512fe3; no voters:
I20260927 14:55:30.280807 19081 leader_election.cc:288] T 00000000000000000000000000000000 P 5917088264224f6ba2219c70f2512fe3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 14:55:30.280915 19086 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5917088264224f6ba2219c70f2512fe3 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 14:55:30.281188 19086 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5917088264224f6ba2219c70f2512fe3 [term 1 LEADER]: Becoming Leader. State: Replica: 5917088264224f6ba2219c70f2512fe3, State: Running, Role: LEADER
I20260927 14:55:30.281502 19086 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5917088264224f6ba2219c70f2512fe3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5917088264224f6ba2219c70f2512fe3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37253 } }
I20260927 14:55:30.281982 19081 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5917088264224f6ba2219c70f2512fe3 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 14:55:30.282966 19088 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5917088264224f6ba2219c70f2512fe3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5917088264224f6ba2219c70f2512fe3. Latest consensus state: current_term: 1 leader_uuid: "5917088264224f6ba2219c70f2512fe3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5917088264224f6ba2219c70f2512fe3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37253 } } }
I20260927 14:55:30.283097 19088 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5917088264224f6ba2219c70f2512fe3 [sys.catalog]: This master's current role is: LEADER
I20260927 14:55:30.283484 19095 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 14:55:30.283452 19087 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5917088264224f6ba2219c70f2512fe3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5917088264224f6ba2219c70f2512fe3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5917088264224f6ba2219c70f2512fe3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37253 } } }
I20260927 14:55:30.283591 19087 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5917088264224f6ba2219c70f2512fe3 [sys.catalog]: This master's current role is: LEADER
I20260927 14:55:30.286511 19095 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 14:55:30.291419 19095 catalog_manager.cc:1347] Generated new cluster ID: a07d9b4ca6ef4ac3ba9d4070e7181e74
I20260927 14:55:30.291481 19095 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 14:55:30.311851 19095 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 14:55:30.313081 19095 catalog_manager.cc:1504] Loading token signing keys...
I20260927 14:55:30.321549 19095 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5917088264224f6ba2219c70f2512fe3: Generated new TSK 0
I20260927 14:55:30.322341 19095 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 14:55:30.334270 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6936m5/build/debug/bin/kudu
/tmp/dist-test-task6936m5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790520684602251-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790520684602251-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790520684602251-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790520684602251-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37253
--builtin_ntp_servers=127.0.98.20:41419
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260927 14:55:30.444720 19105 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 14:55:30.445052 19105 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 14:55:30.445145 19105 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 14:55:30.449280 19105 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 14:55:30.449368 19105 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 14:55:30.449460 19105 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 14:55:30.454308 19105 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41419
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790520684602251-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790520684602251-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790520684602251-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37253
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19105
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790520684602251-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:15:17 UTC on ccycloud.quasar-ledxfe.root.comops.site
build id 4676134
I20260927 14:55:30.455425 19105 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 14:55:30.456790 19105 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 14:55:30.466043 19111 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 14:55:30.466104 19110 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 14:55:30.466142 19113 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 14:55:30.466323 19105 server_base.cc:1034] running on GCE node
I20260927 14:55:30.466774 19105 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 14:55:30.467374 19105 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 14:55:30.468525 19105 hybrid_clock.cc:648] HybridClock initialized: now 1790520930468509 us; error 27 us; skew 500 ppm
I20260927 14:55:30.470427 19105 webserver.cc:460] Webserver started at http://127.0.98.1:39387/ using document root <none> and password file <none>
I20260927 14:55:30.470990 19105 fs_manager.cc:359] Metadata directory not provided
I20260927 14:55:30.471048 19105 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 14:55:30.471290 19105 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 14:55:30.473302 19105 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6936m5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790520684602251-392-0/minicluster-data/ts-0/data/instance:
uuid: "ebb20837f7004e9ab9e294299a648287"
format_stamp: "Formatted at 2026-09-27 14:55:30 on dist-test-slave-xk1g"
server_key: "5aa46cee78fa8f70c9e53b33f4afbfb9"
server_key_iv: "13378b2c6f1ed769292a6b823e80951e"
server_key_version: "encryptionkey@0"
I20260927 14:55:30.473838 19105 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6936m5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790520684602251-392-0/minicluster-data/ts-0/wal/instance:
uuid: "ebb20837f7004e9ab9e294299a648287"
format_stamp: "Formatted at 2026-09-27 14:55:30 on dist-test-slave-xk1g"
server_key: "5aa46cee78fa8f70c9e53b33f4afbfb9"
server_key_iv: "13378b2c6f1ed769292a6b823e80951e"
server_key_version: "encryptionkey@0"
I20260927 14:55:30.477294 19105 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260927 14:55:30.479650 19119 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 14:55:30.480844 19105 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 14:55:30.480978 19105 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790520684602251-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task6936m5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790520684602251-392-0/minicluster-data/ts-0/wal
uuid: "ebb20837f7004e9ab9e294299a648287"
format_stamp: "Formatted at 2026-09-27 14:55:30 on dist-test-slave-xk1g"
server_key: "5aa46cee78fa8f70c9e53b33f4afbfb9"
server_key_iv: "13378b2c6f1ed769292a6b823e80951e"
server_key_version: "encryptionkey@0"
I20260927 14:55:30.481117 19105 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790520684602251-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790520684602251-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790520684602251-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 14:55:30.506865 19105 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 14:55:30.507550 19105 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 14:55:30.507754 19105 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 14:55:30.508328 19105 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 14:55:30.509310 19105 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 14:55:30.509383 19105 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 14:55:30.509454 19105 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 14:55:30.509507 19105 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 14:55:30.518075 19105 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:44413
I20260927 14:55:30.518107 19232 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:44413 every 8 connection(s)
I20260927 14:55:30.519091 19105 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6936m5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790520684602251-392-0/minicluster-data/ts-0/data/info.pb
I20260927 14:55:30.520746 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6936m5/build/debug/bin/kudu as pid 19105
I20260927 14:55:30.520867 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6936m5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790520684602251-392-0/minicluster-data/ts-0/wal/instance
I20260927 14:55:30.521193 392 external_mini_cluster.cc:1428] Setting key 708e46c452d0a55ae3cf1119de859593
I20260927 14:55:30.523491 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6936m5/build/debug/bin/kudu
/tmp/dist-test-task6936m5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790520684602251-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790520684602251-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790520684602251-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790520684602251-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37253
--builtin_ntp_servers=127.0.98.20:41419
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260927 14:55:30.530521 19084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:55:30.521736 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:37909 (local address 127.0.98.62:37253)
0927 14:55:30.522190 (+ 454us) server_negotiation.cc:206] Beginning negotiation
0927 14:55:30.522197 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 14:55:30.522934 (+ 737us) server_negotiation.cc:407] Connection header received
0927 14:55:30.524431 (+ 1497us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:55:30.524434 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:55:30.524485 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:55:30.524564 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:55:30.525675 (+ 1111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:55:30.526535 (+ 860us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:55:30.528420 (+ 1885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:55:30.528637 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:55:30.528727 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:55:30.530063 (+ 1336us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:55:30.530066 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:55:30.530068 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 14:55:30.530081 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:55:30.530098 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 14:55:30.530106 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:55:30.530165 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0927 14:55:30.530359 (+ 194us) server_negotiation.cc:299] Negotiation successful
0927 14:55:30.530408 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":360}
I20260927 14:55:30.531790 19235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:55:30.522016 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37253 (local address 127.0.98.1:37909)
0927 14:55:30.522780 (+ 764us) negotiation.cc:107] Waiting for socket to connect
0927 14:55:30.522834 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0927 14:55:30.524200 (+ 1366us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:55:30.524730 (+ 530us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:55:30.524737 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:55:30.524818 (+ 81us) client_negotiation.cc:189] Negotiated authn=SASL
0927 14:55:30.525520 (+ 702us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:55:30.525536 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:55:30.526703 (+ 1167us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:55:30.526709 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:55:30.528292 (+ 1583us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:55:30.528306 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:55:30.528806 (+ 500us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:55:30.528814 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:55:30.529094 (+ 280us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:55:30.529750 (+ 656us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 14:55:30.529754 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 14:55:30.529815 (+ 61us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 14:55:30.529906 (+ 91us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 14:55:30.530198 (+ 292us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 14:55:30.530209 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 14:55:30.530217 (+ 8us) client_negotiation.cc:769] Sending connection context
0927 14:55:30.530360 (+ 143us) client_negotiation.cc:240] Negotiation successful
0927 14:55:30.530683 (+ 323us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":286,"thread_start_us":135,"threads_started":1}
I20260927 14:55:30.532972 19233 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37253
I20260927 14:55:30.533244 19233 heartbeater.cc:464] Registering TS with master...
I20260927 14:55:30.533766 19233 heartbeater.cc:511] Master 127.0.98.62:37253 requested a full tablet report, sending...
I20260927 14:55:30.535187 19045 ts_manager.cc:194] Registered new tserver with Master: ebb20837f7004e9ab9e294299a648287 (127.0.98.1:44413)
I20260927 14:55:30.536305 19045 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.1:37909
W20260927 14:55:30.633661 19236 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 14:55:30.633963 19236 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 14:55:30.634035 19236 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 14:55:30.638110 19236 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 14:55:30.638192 19236 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 14:55:30.638283 19236 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 14:55:30.643020 19236 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41419
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790520684602251-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790520684602251-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790520684602251-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37253
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19236
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790520684602251-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:15:17 UTC on ccycloud.quasar-ledxfe.root.comops.site
build id 4676134
I20260927 14:55:30.644109 19236 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 14:55:30.645534 19236 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 14:55:30.654680 19242 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 14:55:30.654700 19244 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 14:55:30.654767 19236 server_base.cc:1034] running on GCE node
W20260927 14:55:30.654703 19241 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 14:55:30.655270 19236 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 14:55:30.655922 19236 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 14:55:30.657128 19236 hybrid_clock.cc:648] HybridClock initialized: now 1790520930657114 us; error 24 us; skew 500 ppm
I20260927 14:55:30.658876 19236 webserver.cc:460] Webserver started at http://127.0.98.2:46481/ using document root <none> and password file <none>
I20260927 14:55:30.659442 19236 fs_manager.cc:359] Metadata directory not provided
I20260927 14:55:30.659534 19236 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 14:55:30.659742 19236 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 14:55:30.661777 19236 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6936m5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790520684602251-392-0/minicluster-data/ts-1/data/instance:
uuid: "726c74bbf7d9421d9039dc2e8b32d83c"
format_stamp: "Formatted at 2026-09-27 14:55:30 on dist-test-slave-xk1g"
server_key: "3729103f1b532cdf9d1e23976257d98b"
server_key_iv: "a2f8ae6f1d7cce4802a07b0c70e741f1"
server_key_version: "encryptionkey@0"
I20260927 14:55:30.662326 19236 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6936m5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790520684602251-392-0/minicluster-data/ts-1/wal/instance:
uuid: "726c74bbf7d9421d9039dc2e8b32d83c"
format_stamp: "Formatted at 2026-09-27 14:55:30 on dist-test-slave-xk1g"
server_key: "3729103f1b532cdf9d1e23976257d98b"
server_key_iv: "a2f8ae6f1d7cce4802a07b0c70e741f1"
server_key_version: "encryptionkey@0"
I20260927 14:55:30.665812 19236 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260927 14:55:30.668113 19250 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 14:55:30.669373 19236 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 14:55:30.669492 19236 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790520684602251-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task6936m5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790520684602251-392-0/minicluster-data/ts-1/wal
uuid: "726c74bbf7d9421d9039dc2e8b32d83c"
format_stamp: "Formatted at 2026-09-27 14:55:30 on dist-test-slave-xk1g"
server_key: "3729103f1b532cdf9d1e23976257d98b"
server_key_iv: "a2f8ae6f1d7cce4802a07b0c70e741f1"
server_key_version: "encryptionkey@0"
I20260927 14:55:30.669600 19236 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790520684602251-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790520684602251-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790520684602251-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 14:55:30.698987 19236 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 14:55:30.699646 19236 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 14:55:30.699805 19236 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 14:55:30.700328 19236 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 14:55:30.701349 19236 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 14:55:30.701396 19236 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 14:55:30.701434 19236 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 14:55:30.701455 19236 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 14:55:30.710140 19236 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:41977
I20260927 14:55:30.710192 19363 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:41977 every 8 connection(s)
I20260927 14:55:30.711139 19236 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6936m5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790520684602251-392-0/minicluster-data/ts-1/data/info.pb
I20260927 14:55:30.720230 19084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:55:30.712989 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:37657 (local address 127.0.98.62:37253)
0927 14:55:30.713141 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0927 14:55:30.713145 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 14:55:30.714192 (+ 1047us) server_negotiation.cc:407] Connection header received
0927 14:55:30.715594 (+ 1402us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:55:30.715597 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:55:30.715673 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:55:30.715773 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:55:30.716661 (+ 888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:55:30.717247 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:55:30.718388 (+ 1141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:55:30.718613 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:55:30.718723 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:55:30.719741 (+ 1018us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:55:30.719745 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:55:30.719748 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 14:55:30.719766 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:55:30.719783 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 14:55:30.719790 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:55:30.719865 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0927 14:55:30.720027 (+ 162us) server_negotiation.cc:299] Negotiation successful
0927 14:55:30.720087 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260927 14:55:30.720906 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6936m5/build/debug/bin/kudu as pid 19236
I20260927 14:55:30.721029 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6936m5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790520684602251-392-0/minicluster-data/ts-1/wal/instance
I20260927 14:55:30.721343 392 external_mini_cluster.cc:1428] Setting key 1d033a15317906f5b73409bd487df3a1
I20260927 14:55:30.721328 19366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:55:30.713408 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37253 (local address 127.0.98.2:37657)
0927 14:55:30.714033 (+ 625us) negotiation.cc:107] Waiting for socket to connect
0927 14:55:30.714074 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0927 14:55:30.715378 (+ 1304us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:55:30.715945 (+ 567us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:55:30.715952 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:55:30.716035 (+ 83us) client_negotiation.cc:189] Negotiated authn=SASL
0927 14:55:30.716513 (+ 478us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:55:30.716525 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:55:30.717370 (+ 845us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:55:30.717373 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:55:30.718257 (+ 884us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:55:30.718265 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:55:30.718773 (+ 508us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:55:30.718778 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:55:30.718932 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:55:30.719495 (+ 563us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 14:55:30.719496 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0927 14:55:30.719534 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 14:55:30.719593 (+ 59us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 14:55:30.719878 (+ 285us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 14:55:30.719890 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 14:55:30.719899 (+ 9us) client_negotiation.cc:769] Sending connection context
0927 14:55:30.720015 (+ 116us) client_negotiation.cc:240] Negotiation successful
0927 14:55:30.720299 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":244,"thread_start_us":100,"threads_started":1}
I20260927 14:55:30.722513 19364 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37253
I20260927 14:55:30.722769 19364 heartbeater.cc:464] Registering TS with master...
I20260927 14:55:30.723362 19364 heartbeater.cc:511] Master 127.0.98.62:37253 requested a full tablet report, sending...
I20260927 14:55:30.723510 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6936m5/build/debug/bin/kudu
/tmp/dist-test-task6936m5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790520684602251-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790520684602251-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790520684602251-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790520684602251-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37253
--builtin_ntp_servers=127.0.98.20:41419
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260927 14:55:30.724452 19044 ts_manager.cc:194] Registered new tserver with Master: 726c74bbf7d9421d9039dc2e8b32d83c (127.0.98.2:41977)
I20260927 14:55:30.725332 19044 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.2:37657
W20260927 14:55:30.834406 19367 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 14:55:30.834651 19367 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 14:55:30.834687 19367 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 14:55:30.838650 19367 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 14:55:30.838697 19367 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 14:55:30.838778 19367 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 14:55:30.843966 19367 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41419
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790520684602251-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790520684602251-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790520684602251-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37253
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19367
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790520684602251-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:15:17 UTC on ccycloud.quasar-ledxfe.root.comops.site
build id 4676134
I20260927 14:55:30.844976 19367 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 14:55:30.846310 19367 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 14:55:30.855324 19375 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 14:55:30.855324 19373 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 14:55:30.855355 19372 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 14:55:30.855391 19367 server_base.cc:1034] running on GCE node
I20260927 14:55:30.855983 19367 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 14:55:30.856557 19367 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 14:55:30.857729 19367 hybrid_clock.cc:648] HybridClock initialized: now 1790520930857717 us; error 22 us; skew 500 ppm
I20260927 14:55:30.859480 19367 webserver.cc:460] Webserver started at http://127.0.98.3:44407/ using document root <none> and password file <none>
I20260927 14:55:30.860040 19367 fs_manager.cc:359] Metadata directory not provided
I20260927 14:55:30.860123 19367 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 14:55:30.860337 19367 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 14:55:30.862286 19367 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6936m5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790520684602251-392-0/minicluster-data/ts-2/data/instance:
uuid: "da6f360f32fe4154b7d5282f8b627f48"
format_stamp: "Formatted at 2026-09-27 14:55:30 on dist-test-slave-xk1g"
server_key: "9d28744c1d6c07d33ebc54a00234bef1"
server_key_iv: "5c5e6d12e561e1867fd51ab7b20ae880"
server_key_version: "encryptionkey@0"
I20260927 14:55:30.862856 19367 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6936m5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790520684602251-392-0/minicluster-data/ts-2/wal/instance:
uuid: "da6f360f32fe4154b7d5282f8b627f48"
format_stamp: "Formatted at 2026-09-27 14:55:30 on dist-test-slave-xk1g"
server_key: "9d28744c1d6c07d33ebc54a00234bef1"
server_key_iv: "5c5e6d12e561e1867fd51ab7b20ae880"
server_key_version: "encryptionkey@0"
I20260927 14:55:30.866216 19367 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 14:55:30.868402 19381 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 14:55:30.869637 19367 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 14:55:30.869812 19367 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790520684602251-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task6936m5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790520684602251-392-0/minicluster-data/ts-2/wal
uuid: "da6f360f32fe4154b7d5282f8b627f48"
format_stamp: "Formatted at 2026-09-27 14:55:30 on dist-test-slave-xk1g"
server_key: "9d28744c1d6c07d33ebc54a00234bef1"
server_key_iv: "5c5e6d12e561e1867fd51ab7b20ae880"
server_key_version: "encryptionkey@0"
I20260927 14:55:30.869920 19367 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790520684602251-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790520684602251-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790520684602251-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 14:55:30.884740 19367 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 14:55:30.885443 19367 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 14:55:30.885613 19367 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 14:55:30.886129 19367 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 14:55:30.886951 19367 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 14:55:30.886996 19367 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 14:55:30.887033 19367 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 14:55:30.887054 19367 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 14:55:30.895166 19367 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:41091
I20260927 14:55:30.895210 19494 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:41091 every 8 connection(s)
I20260927 14:55:30.896026 19367 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6936m5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790520684602251-392-0/minicluster-data/ts-2/data/info.pb
I20260927 14:55:30.900871 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6936m5/build/debug/bin/kudu as pid 19367
I20260927 14:55:30.900960 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6936m5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790520684602251-392-0/minicluster-data/ts-2/wal/instance
I20260927 14:55:30.901252 392 external_mini_cluster.cc:1428] Setting key b7025e6637462df914967e8a281e94db
I20260927 14:55:30.905442 19084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:55:30.897776 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:43011 (local address 127.0.98.62:37253)
0927 14:55:30.897921 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0927 14:55:30.897926 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 14:55:30.898968 (+ 1042us) server_negotiation.cc:407] Connection header received
0927 14:55:30.900281 (+ 1313us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:55:30.900284 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:55:30.900331 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:55:30.900410 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:55:30.901448 (+ 1038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:55:30.902125 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:55:30.903468 (+ 1343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:55:30.903688 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:55:30.903812 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:55:30.904923 (+ 1111us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:55:30.904926 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:55:30.904929 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 14:55:30.904949 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:55:30.904968 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 14:55:30.904976 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:55:30.905058 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0927 14:55:30.905245 (+ 187us) server_negotiation.cc:299] Negotiation successful
0927 14:55:30.905309 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260927 14:55:30.906262 19497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:55:30.898214 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37253 (local address 127.0.98.3:43011)
0927 14:55:30.898824 (+ 610us) negotiation.cc:107] Waiting for socket to connect
0927 14:55:30.898863 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0927 14:55:30.900079 (+ 1216us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:55:30.900566 (+ 487us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:55:30.900572 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:55:30.900656 (+ 84us) client_negotiation.cc:189] Negotiated authn=SASL
0927 14:55:30.901298 (+ 642us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:55:30.901321 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:55:30.902234 (+ 913us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:55:30.902238 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:55:30.903349 (+ 1111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:55:30.903359 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:55:30.903851 (+ 492us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:55:30.903855 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:55:30.904030 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:55:30.904668 (+ 638us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 14:55:30.904670 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0927 14:55:30.904709 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 14:55:30.904771 (+ 62us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 14:55:30.905083 (+ 312us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 14:55:30.905094 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 14:55:30.905099 (+ 5us) client_negotiation.cc:769] Sending connection context
0927 14:55:30.905235 (+ 136us) client_negotiation.cc:240] Negotiation successful
0927 14:55:30.905520 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":241,"thread_start_us":108,"threads_started":1}
I20260927 14:55:30.907246 19495 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37253
I20260927 14:55:30.907478 19495 heartbeater.cc:464] Registering TS with master...
I20260927 14:55:30.907927 19495 heartbeater.cc:511] Master 127.0.98.62:37253 requested a full tablet report, sending...
I20260927 14:55:30.908811 19044 ts_manager.cc:194] Registered new tserver with Master: da6f360f32fe4154b7d5282f8b627f48 (127.0.98.3:41091)
I20260927 14:55:30.909416 19044 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.3:43011
I20260927 14:55:30.914796 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 14:55:30.920019 19084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:55:30.916587 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44596 (local address 127.0.98.62:37253)
0927 14:55:30.916734 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0927 14:55:30.916740 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 14:55:30.916855 (+ 115us) server_negotiation.cc:407] Connection header received
0927 14:55:30.916948 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:55:30.916950 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:55:30.916987 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:55:30.917093 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:55:30.917325 (+ 232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:55:30.917813 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:55:30.918587 (+ 774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:55:30.918792 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:55:30.918864 (+ 72us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:55:30.919573 (+ 709us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:55:30.919576 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:55:30.919578 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 14:55:30.919589 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:55:30.919605 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 14:55:30.919612 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:55:30.919709 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0927 14:55:30.919804 (+ 95us) server_negotiation.cc:299] Negotiation successful
0927 14:55:30.919858 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260927 14:55:30.922721 19044 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44596:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 14:55:30.925626 19044 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 14:55:30.939602 19507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:55:30.933939 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37084 (local address 127.0.98.3:41091)
0927 14:55:30.934318 (+ 379us) server_negotiation.cc:206] Beginning negotiation
0927 14:55:30.934329 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 14:55:30.934601 (+ 272us) server_negotiation.cc:407] Connection header received
0927 14:55:30.934696 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:55:30.934704 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:55:30.934922 (+ 218us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:55:30.935728 (+ 806us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 14:55:30.935889 (+ 161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:55:30.936695 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:55:30.938309 (+ 1614us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:55:30.938952 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:55:30.939076 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:55:30.939290 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0927 14:55:30.939346 (+ 56us) server_negotiation.cc:299] Negotiation successful
0927 14:55:30.939437 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":290,"thread_start_us":89,"threads_started":1}
I20260927 14:55:30.939599 19506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:55:30.933771 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:41091 (local address 127.0.0.1:37084)
0927 14:55:30.934146 (+ 375us) negotiation.cc:107] Waiting for socket to connect
0927 14:55:30.934196 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0927 14:55:30.934559 (+ 363us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:55:30.935087 (+ 528us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:55:30.935093 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:55:30.935113 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 14:55:30.935577 (+ 464us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:55:30.935584 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:55:30.936811 (+ 1227us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:55:30.936815 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:55:30.938181 (+ 1366us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:55:30.938191 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:55:30.939075 (+ 884us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:55:30.939078 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:55:30.939187 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:55:30.939222 (+ 35us) client_negotiation.cc:769] Sending connection context
0927 14:55:30.939322 (+ 100us) client_negotiation.cc:240] Negotiation successful
0927 14:55:30.939437 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":184,"mutex_wait_us":25,"thread_start_us":94,"threads_started":1}
I20260927 14:55:30.940039 19509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:55:30.934509 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:41977 (local address 127.0.0.1:47568)
0927 14:55:30.935261 (+ 752us) negotiation.cc:107] Waiting for socket to connect
0927 14:55:30.935276 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 14:55:30.935361 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:55:30.935861 (+ 500us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:55:30.935864 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:55:30.935883 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 14:55:30.935976 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:55:30.935982 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:55:30.937311 (+ 1329us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:55:30.937314 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:55:30.938453 (+ 1139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:55:30.938464 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:55:30.939684 (+ 1220us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:55:30.939687 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:55:30.939761 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:55:30.939781 (+ 20us) client_negotiation.cc:769] Sending connection context
0927 14:55:30.939848 (+ 67us) client_negotiation.cc:240] Negotiation successful
0927 14:55:30.939912 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":694,"mutex_wait_us":3,"thread_start_us":90,"threads_started":1}
I20260927 14:55:30.940167 19510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:55:30.934584 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34120 (local address 127.0.98.1:44413)
0927 14:55:30.935021 (+ 437us) server_negotiation.cc:206] Beginning negotiation
0927 14:55:30.935033 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 14:55:30.935057 (+ 24us) server_negotiation.cc:407] Connection header received
0927 14:55:30.935130 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:55:30.935136 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:55:30.935323 (+ 187us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:55:30.935556 (+ 233us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 14:55:30.935786 (+ 230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:55:30.936987 (+ 1201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:55:30.938434 (+ 1447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:55:30.939223 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:55:30.939693 (+ 470us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:55:30.939868 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0927 14:55:30.939915 (+ 47us) server_negotiation.cc:299] Negotiation successful
0927 14:55:30.940003 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":332,"thread_start_us":107,"threads_started":1}
I20260927 14:55:30.940516 19511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:55:30.934911 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47568 (local address 127.0.98.2:41977)
0927 14:55:30.935276 (+ 365us) server_negotiation.cc:206] Beginning negotiation
0927 14:55:30.935287 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 14:55:30.935346 (+ 59us) server_negotiation.cc:407] Connection header received
0927 14:55:30.935510 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:55:30.935515 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:55:30.935700 (+ 185us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:55:30.935871 (+ 171us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 14:55:30.936100 (+ 229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:55:30.937190 (+ 1090us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:55:30.938579 (+ 1389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:55:30.939352 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:55:30.939443 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:55:30.939623 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0927 14:55:30.940286 (+ 663us) server_negotiation.cc:299] Negotiation successful
0927 14:55:30.940384 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":270,"thread_start_us":86,"threads_started":1}
I20260927 14:55:30.940846 19508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:55:30.934141 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44413 (local address 127.0.0.1:34120)
0927 14:55:30.934809 (+ 668us) negotiation.cc:107] Waiting for socket to connect
0927 14:55:30.934822 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 14:55:30.934891 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:55:30.935549 (+ 658us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:55:30.935553 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:55:30.935569 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 14:55:30.935662 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:55:30.935671 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:55:30.937123 (+ 1452us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:55:30.937126 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:55:30.938304 (+ 1178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:55:30.938321 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:55:30.939366 (+ 1045us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:55:30.939372 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:55:30.939462 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:55:30.939483 (+ 21us) client_negotiation.cc:769] Sending connection context
0927 14:55:30.939530 (+ 47us) client_negotiation.cc:240] Negotiation successful
0927 14:55:30.939597 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":609,"mutex_wait_us":46,"thread_start_us":104,"threads_started":1}
I20260927 14:55:30.943140 19298 tablet_service.cc:1467] Processing CreateTablet for tablet 4305db92f72a412eb1943067f0f8fc7f (DEFAULT_TABLE table=test-table [id=c54e0e7604bd4bccacb9a82006024334]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 14:55:30.943143 19429 tablet_service.cc:1467] Processing CreateTablet for tablet 4305db92f72a412eb1943067f0f8fc7f (DEFAULT_TABLE table=test-table [id=c54e0e7604bd4bccacb9a82006024334]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 14:55:30.943511 19167 tablet_service.cc:1467] Processing CreateTablet for tablet 4305db92f72a412eb1943067f0f8fc7f (DEFAULT_TABLE table=test-table [id=c54e0e7604bd4bccacb9a82006024334]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 14:55:30.944151 19298 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4305db92f72a412eb1943067f0f8fc7f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 14:55:30.944164 19429 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4305db92f72a412eb1943067f0f8fc7f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 14:55:30.944392 19167 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4305db92f72a412eb1943067f0f8fc7f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 14:55:30.950445 19512 tablet_bootstrap.cc:492] T 4305db92f72a412eb1943067f0f8fc7f P 726c74bbf7d9421d9039dc2e8b32d83c: Bootstrap starting.
I20260927 14:55:30.951692 19514 tablet_bootstrap.cc:492] T 4305db92f72a412eb1943067f0f8fc7f P da6f360f32fe4154b7d5282f8b627f48: Bootstrap starting.
I20260927 14:55:30.953151 19512 tablet_bootstrap.cc:654] T 4305db92f72a412eb1943067f0f8fc7f P 726c74bbf7d9421d9039dc2e8b32d83c: Neither blocks nor log segments found. Creating new log.
I20260927 14:55:30.953447 19514 tablet_bootstrap.cc:654] T 4305db92f72a412eb1943067f0f8fc7f P da6f360f32fe4154b7d5282f8b627f48: Neither blocks nor log segments found. Creating new log.
I20260927 14:55:30.953938 19512 log.cc:824] T 4305db92f72a412eb1943067f0f8fc7f P 726c74bbf7d9421d9039dc2e8b32d83c: Log is configured to *not* fsync() on all Append() calls
I20260927 14:55:30.954154 19514 log.cc:824] T 4305db92f72a412eb1943067f0f8fc7f P da6f360f32fe4154b7d5282f8b627f48: Log is configured to *not* fsync() on all Append() calls
I20260927 14:55:30.954646 19513 tablet_bootstrap.cc:492] T 4305db92f72a412eb1943067f0f8fc7f P ebb20837f7004e9ab9e294299a648287: Bootstrap starting.
I20260927 14:55:30.955965 19512 tablet_bootstrap.cc:492] T 4305db92f72a412eb1943067f0f8fc7f P 726c74bbf7d9421d9039dc2e8b32d83c: No bootstrap required, opened a new log
I20260927 14:55:30.955961 19514 tablet_bootstrap.cc:492] T 4305db92f72a412eb1943067f0f8fc7f P da6f360f32fe4154b7d5282f8b627f48: No bootstrap required, opened a new log
I20260927 14:55:30.956148 19514 ts_tablet_manager.cc:1397] T 4305db92f72a412eb1943067f0f8fc7f P da6f360f32fe4154b7d5282f8b627f48: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260927 14:55:30.956148 19512 ts_tablet_manager.cc:1397] T 4305db92f72a412eb1943067f0f8fc7f P 726c74bbf7d9421d9039dc2e8b32d83c: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.003s
I20260927 14:55:30.956946 19513 tablet_bootstrap.cc:654] T 4305db92f72a412eb1943067f0f8fc7f P ebb20837f7004e9ab9e294299a648287: Neither blocks nor log segments found. Creating new log.
I20260927 14:55:30.957774 19513 log.cc:824] T 4305db92f72a412eb1943067f0f8fc7f P ebb20837f7004e9ab9e294299a648287: Log is configured to *not* fsync() on all Append() calls
I20260927 14:55:30.959494 19513 tablet_bootstrap.cc:492] T 4305db92f72a412eb1943067f0f8fc7f P ebb20837f7004e9ab9e294299a648287: No bootstrap required, opened a new log
I20260927 14:55:30.959712 19513 ts_tablet_manager.cc:1397] T 4305db92f72a412eb1943067f0f8fc7f P ebb20837f7004e9ab9e294299a648287: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260927 14:55:30.960578 19514 raft_consensus.cc:348] T 4305db92f72a412eb1943067f0f8fc7f P da6f360f32fe4154b7d5282f8b627f48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da6f360f32fe4154b7d5282f8b627f48" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41091 } } peers { permanent_uuid: "726c74bbf7d9421d9039dc2e8b32d83c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41977 } } peers { permanent_uuid: "ebb20837f7004e9ab9e294299a648287" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44413 } }
I20260927 14:55:30.960573 19512 raft_consensus.cc:348] T 4305db92f72a412eb1943067f0f8fc7f P 726c74bbf7d9421d9039dc2e8b32d83c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da6f360f32fe4154b7d5282f8b627f48" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41091 } } peers { permanent_uuid: "726c74bbf7d9421d9039dc2e8b32d83c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41977 } } peers { permanent_uuid: "ebb20837f7004e9ab9e294299a648287" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44413 } }
I20260927 14:55:30.960835 19514 raft_consensus.cc:374] T 4305db92f72a412eb1943067f0f8fc7f P da6f360f32fe4154b7d5282f8b627f48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 14:55:30.960835 19512 raft_consensus.cc:374] T 4305db92f72a412eb1943067f0f8fc7f P 726c74bbf7d9421d9039dc2e8b32d83c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 14:55:30.960892 19512 raft_consensus.cc:729] T 4305db92f72a412eb1943067f0f8fc7f P 726c74bbf7d9421d9039dc2e8b32d83c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 726c74bbf7d9421d9039dc2e8b32d83c, State: Initialized, Role: FOLLOWER
I20260927 14:55:30.960892 19514 raft_consensus.cc:729] T 4305db92f72a412eb1943067f0f8fc7f P da6f360f32fe4154b7d5282f8b627f48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: da6f360f32fe4154b7d5282f8b627f48, State: Initialized, Role: FOLLOWER
I20260927 14:55:30.961362 19512 consensus_queue.cc:260] T 4305db92f72a412eb1943067f0f8fc7f P 726c74bbf7d9421d9039dc2e8b32d83c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da6f360f32fe4154b7d5282f8b627f48" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41091 } } peers { permanent_uuid: "726c74bbf7d9421d9039dc2e8b32d83c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41977 } } peers { permanent_uuid: "ebb20837f7004e9ab9e294299a648287" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44413 } }
I20260927 14:55:30.961362 19514 consensus_queue.cc:260] T 4305db92f72a412eb1943067f0f8fc7f P da6f360f32fe4154b7d5282f8b627f48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da6f360f32fe4154b7d5282f8b627f48" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41091 } } peers { permanent_uuid: "726c74bbf7d9421d9039dc2e8b32d83c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41977 } } peers { permanent_uuid: "ebb20837f7004e9ab9e294299a648287" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44413 } }
I20260927 14:55:30.962318 19514 ts_tablet_manager.cc:1428] T 4305db92f72a412eb1943067f0f8fc7f P da6f360f32fe4154b7d5282f8b627f48: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260927 14:55:30.962425 19512 ts_tablet_manager.cc:1428] T 4305db92f72a412eb1943067f0f8fc7f P 726c74bbf7d9421d9039dc2e8b32d83c: Time spent starting tablet: real 0.006s user 0.006s sys 0.001s
I20260927 14:55:30.962716 19364 heartbeater.cc:503] Master 127.0.98.62:37253 was elected leader, sending a full tablet report...
I20260927 14:55:30.962833 19495 heartbeater.cc:503] Master 127.0.98.62:37253 was elected leader, sending a full tablet report...
I20260927 14:55:30.965140 19513 raft_consensus.cc:348] T 4305db92f72a412eb1943067f0f8fc7f P ebb20837f7004e9ab9e294299a648287 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da6f360f32fe4154b7d5282f8b627f48" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41091 } } peers { permanent_uuid: "726c74bbf7d9421d9039dc2e8b32d83c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41977 } } peers { permanent_uuid: "ebb20837f7004e9ab9e294299a648287" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44413 } }
I20260927 14:55:30.965372 19513 raft_consensus.cc:374] T 4305db92f72a412eb1943067f0f8fc7f P ebb20837f7004e9ab9e294299a648287 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 14:55:30.965438 19513 raft_consensus.cc:729] T 4305db92f72a412eb1943067f0f8fc7f P ebb20837f7004e9ab9e294299a648287 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ebb20837f7004e9ab9e294299a648287, State: Initialized, Role: FOLLOWER
I20260927 14:55:30.965910 19513 consensus_queue.cc:260] T 4305db92f72a412eb1943067f0f8fc7f P ebb20837f7004e9ab9e294299a648287 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da6f360f32fe4154b7d5282f8b627f48" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41091 } } peers { permanent_uuid: "726c74bbf7d9421d9039dc2e8b32d83c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41977 } } peers { permanent_uuid: "ebb20837f7004e9ab9e294299a648287" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44413 } }
I20260927 14:55:30.966941 19513 ts_tablet_manager.cc:1428] T 4305db92f72a412eb1943067f0f8fc7f P ebb20837f7004e9ab9e294299a648287: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260927 14:55:30.967026 19233 heartbeater.cc:503] Master 127.0.98.62:37253 was elected leader, sending a full tablet report...
I20260927 14:55:30.967208 19519 raft_consensus.cc:482] T 4305db92f72a412eb1943067f0f8fc7f P da6f360f32fe4154b7d5282f8b627f48 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 14:55:30.967337 19519 raft_consensus.cc:504] T 4305db92f72a412eb1943067f0f8fc7f P da6f360f32fe4154b7d5282f8b627f48 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da6f360f32fe4154b7d5282f8b627f48" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41091 } } peers { permanent_uuid: "726c74bbf7d9421d9039dc2e8b32d83c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41977 } } peers { permanent_uuid: "ebb20837f7004e9ab9e294299a648287" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44413 } }
I20260927 14:55:30.968331 19519 leader_election.cc:288] T 4305db92f72a412eb1943067f0f8fc7f P da6f360f32fe4154b7d5282f8b627f48 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 726c74bbf7d9421d9039dc2e8b32d83c (127.0.98.2:41977), ebb20837f7004e9ab9e294299a648287 (127.0.98.1:44413)
I20260927 14:55:30.972818 19497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:55:30.968519 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:41977 (local address 127.0.98.3:45639)
0927 14:55:30.968790 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0927 14:55:30.968805 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 14:55:30.968884 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:55:30.969448 (+ 564us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:55:30.969451 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:55:30.969474 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 14:55:30.969647 (+ 173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:55:30.969654 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:55:30.970510 (+ 856us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:55:30.970514 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:55:30.971536 (+ 1022us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:55:30.971545 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:55:30.972372 (+ 827us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:55:30.972375 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:55:30.972477 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:55:30.972504 (+ 27us) client_negotiation.cc:769] Sending connection context
0927 14:55:30.972590 (+ 86us) client_negotiation.cc:240] Negotiation successful
0927 14:55:30.972665 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":209,"spinlock_wait_cycles":46976}
I20260927 14:55:30.972848 19511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:55:30.968993 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:45639 (local address 127.0.98.2:41977)
0927 14:55:30.969154 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0927 14:55:30.969159 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 14:55:30.969175 (+ 16us) server_negotiation.cc:407] Connection header received
0927 14:55:30.969252 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:55:30.969256 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:55:30.969320 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:55:30.969392 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 14:55:30.969778 (+ 386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:55:30.970366 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:55:30.971666 (+ 1300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:55:30.972216 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:55:30.972374 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:55:30.972422 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0927 14:55:30.972626 (+ 204us) server_negotiation.cc:299] Negotiation successful
0927 14:55:30.972701 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260927 14:55:30.973155 19510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:55:30.968646 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:42879 (local address 127.0.98.1:44413)
0927 14:55:30.968781 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0927 14:55:30.968785 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 14:55:30.969036 (+ 251us) server_negotiation.cc:407] Connection header received
0927 14:55:30.969641 (+ 605us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:55:30.969644 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:55:30.969690 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:55:30.969778 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 14:55:30.970079 (+ 301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:55:30.970759 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:55:30.972006 (+ 1247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:55:30.972547 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:55:30.972641 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:55:30.972689 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0927 14:55:30.972909 (+ 220us) server_negotiation.cc:299] Negotiation successful
0927 14:55:30.972987 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260927 14:55:30.973163 19521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:55:30.968556 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44413 (local address 127.0.98.3:42879)
0927 14:55:30.968931 (+ 375us) negotiation.cc:107] Waiting for socket to connect
0927 14:55:30.968947 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 14:55:30.969522 (+ 575us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:55:30.969872 (+ 350us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:55:30.969874 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:55:30.969886 (+ 12us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 14:55:30.969973 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:55:30.969977 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:55:30.970879 (+ 902us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:55:30.970882 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:55:30.971869 (+ 987us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:55:30.971881 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:55:30.972660 (+ 779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:55:30.972664 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:55:30.972766 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:55:30.972791 (+ 25us) client_negotiation.cc:769] Sending connection context
0927 14:55:30.972883 (+ 92us) client_negotiation.cc:240] Negotiation successful
0927 14:55:30.972958 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":292,"thread_start_us":108,"threads_started":1}
I20260927 14:55:30.973713 19318 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4305db92f72a412eb1943067f0f8fc7f" candidate_uuid: "da6f360f32fe4154b7d5282f8b627f48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "726c74bbf7d9421d9039dc2e8b32d83c" is_pre_election: true
I20260927 14:55:30.973820 19187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4305db92f72a412eb1943067f0f8fc7f" candidate_uuid: "da6f360f32fe4154b7d5282f8b627f48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ebb20837f7004e9ab9e294299a648287" is_pre_election: true
I20260927 14:55:30.973925 19318 raft_consensus.cc:2454] T 4305db92f72a412eb1943067f0f8fc7f P 726c74bbf7d9421d9039dc2e8b32d83c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate da6f360f32fe4154b7d5282f8b627f48 in term 0.
I20260927 14:55:30.974119 19187 raft_consensus.cc:2454] T 4305db92f72a412eb1943067f0f8fc7f P ebb20837f7004e9ab9e294299a648287 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate da6f360f32fe4154b7d5282f8b627f48 in term 0.
I20260927 14:55:30.974399 19383 leader_election.cc:302] T 4305db92f72a412eb1943067f0f8fc7f P da6f360f32fe4154b7d5282f8b627f48 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 726c74bbf7d9421d9039dc2e8b32d83c, da6f360f32fe4154b7d5282f8b627f48; no voters:
I20260927 14:55:30.974658 19519 raft_consensus.cc:2781] T 4305db92f72a412eb1943067f0f8fc7f P da6f360f32fe4154b7d5282f8b627f48 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 14:55:30.974764 19519 raft_consensus.cc:482] T 4305db92f72a412eb1943067f0f8fc7f P da6f360f32fe4154b7d5282f8b627f48 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 14:55:30.974823 19519 raft_consensus.cc:3037] T 4305db92f72a412eb1943067f0f8fc7f P da6f360f32fe4154b7d5282f8b627f48 [term 0 FOLLOWER]: Advancing to term 1
I20260927 14:55:30.975847 19519 raft_consensus.cc:504] T 4305db92f72a412eb1943067f0f8fc7f P da6f360f32fe4154b7d5282f8b627f48 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da6f360f32fe4154b7d5282f8b627f48" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41091 } } peers { permanent_uuid: "726c74bbf7d9421d9039dc2e8b32d83c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41977 } } peers { permanent_uuid: "ebb20837f7004e9ab9e294299a648287" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44413 } }
I20260927 14:55:30.976191 19519 leader_election.cc:288] T 4305db92f72a412eb1943067f0f8fc7f P da6f360f32fe4154b7d5282f8b627f48 [CANDIDATE]: Term 1 election: Requested vote from peers 726c74bbf7d9421d9039dc2e8b32d83c (127.0.98.2:41977), ebb20837f7004e9ab9e294299a648287 (127.0.98.1:44413)
I20260927 14:55:30.976511 19187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4305db92f72a412eb1943067f0f8fc7f" candidate_uuid: "da6f360f32fe4154b7d5282f8b627f48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ebb20837f7004e9ab9e294299a648287"
I20260927 14:55:30.976488 19318 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4305db92f72a412eb1943067f0f8fc7f" candidate_uuid: "da6f360f32fe4154b7d5282f8b627f48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "726c74bbf7d9421d9039dc2e8b32d83c"
I20260927 14:55:30.976641 19187 raft_consensus.cc:3037] T 4305db92f72a412eb1943067f0f8fc7f P ebb20837f7004e9ab9e294299a648287 [term 0 FOLLOWER]: Advancing to term 1
I20260927 14:55:30.976660 19318 raft_consensus.cc:3037] T 4305db92f72a412eb1943067f0f8fc7f P 726c74bbf7d9421d9039dc2e8b32d83c [term 0 FOLLOWER]: Advancing to term 1
I20260927 14:55:30.977753 19187 raft_consensus.cc:2454] T 4305db92f72a412eb1943067f0f8fc7f P ebb20837f7004e9ab9e294299a648287 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate da6f360f32fe4154b7d5282f8b627f48 in term 1.
I20260927 14:55:30.977752 19318 raft_consensus.cc:2454] T 4305db92f72a412eb1943067f0f8fc7f P 726c74bbf7d9421d9039dc2e8b32d83c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate da6f360f32fe4154b7d5282f8b627f48 in term 1.
I20260927 14:55:30.978053 19382 leader_election.cc:302] T 4305db92f72a412eb1943067f0f8fc7f P da6f360f32fe4154b7d5282f8b627f48 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: da6f360f32fe4154b7d5282f8b627f48, ebb20837f7004e9ab9e294299a648287; no voters:
I20260927 14:55:30.978247 19519 raft_consensus.cc:2781] T 4305db92f72a412eb1943067f0f8fc7f P da6f360f32fe4154b7d5282f8b627f48 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 14:55:30.978444 19519 raft_consensus.cc:686] T 4305db92f72a412eb1943067f0f8fc7f P da6f360f32fe4154b7d5282f8b627f48 [term 1 LEADER]: Becoming Leader. State: Replica: da6f360f32fe4154b7d5282f8b627f48, State: Running, Role: LEADER
I20260927 14:55:30.978734 19519 consensus_queue.cc:237] T 4305db92f72a412eb1943067f0f8fc7f P da6f360f32fe4154b7d5282f8b627f48 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da6f360f32fe4154b7d5282f8b627f48" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41091 } } peers { permanent_uuid: "726c74bbf7d9421d9039dc2e8b32d83c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41977 } } peers { permanent_uuid: "ebb20837f7004e9ab9e294299a648287" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44413 } }
I20260927 14:55:30.981741 19045 catalog_manager.cc:5495] T 4305db92f72a412eb1943067f0f8fc7f P da6f360f32fe4154b7d5282f8b627f48 reported cstate change: term changed from 0 to 1, leader changed from <none> to da6f360f32fe4154b7d5282f8b627f48 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "da6f360f32fe4154b7d5282f8b627f48" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da6f360f32fe4154b7d5282f8b627f48" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41091 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "726c74bbf7d9421d9039dc2e8b32d83c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41977 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ebb20837f7004e9ab9e294299a648287" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44413 } health_report { overall_health: UNKNOWN } } }
I20260927 14:55:31.002620 19507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:55:30.999273 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37092 (local address 127.0.98.3:41091)
0927 14:55:30.999410 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0927 14:55:30.999414 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 14:55:30.999428 (+ 14us) server_negotiation.cc:407] Connection header received
0927 14:55:30.999505 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:55:30.999507 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:55:30.999591 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:55:30.999704 (+ 113us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 14:55:30.999917 (+ 213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:55:31.000414 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:55:31.001306 (+ 892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:55:31.001526 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:55:31.001620 (+ 94us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:55:31.001866 (+ 246us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 14:55:31.002281 (+ 415us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 14:55:31.002394 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0927 14:55:31.002479 (+ 85us) server_negotiation.cc:299] Negotiation successful
0927 14:55:31.002511 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260927 14:55:31.008291 19318 raft_consensus.cc:1261] T 4305db92f72a412eb1943067f0f8fc7f P 726c74bbf7d9421d9039dc2e8b32d83c [term 1 FOLLOWER]: Refusing update from remote peer da6f360f32fe4154b7d5282f8b627f48: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 14:55:31.008299 19187 raft_consensus.cc:1261] T 4305db92f72a412eb1943067f0f8fc7f P ebb20837f7004e9ab9e294299a648287 [term 1 FOLLOWER]: Refusing update from remote peer da6f360f32fe4154b7d5282f8b627f48: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 14:55:31.008970 19522 consensus_queue.cc:1035] T 4305db92f72a412eb1943067f0f8fc7f P da6f360f32fe4154b7d5282f8b627f48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "726c74bbf7d9421d9039dc2e8b32d83c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41977 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 14:55:31.009130 19519 consensus_queue.cc:1035] T 4305db92f72a412eb1943067f0f8fc7f P da6f360f32fe4154b7d5282f8b627f48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ebb20837f7004e9ab9e294299a648287" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44413 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 14:55:31.014889 19525 mvcc.cc:205] Tried to move back new op lower bound from 7333973733403160576 to 7333973733291323392. Current Snapshot: MvccSnapshot[applied={T|T < 7333973733403160576}]
I20260927 14:55:31.026563 19043 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:44596:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 14:55:31.026798 19043 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:44596:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 14:55:31.029451 19043 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5917088264224f6ba2219c70f2512fe3: Sending DeleteTablet for 3 replicas of tablet 4305db92f72a412eb1943067f0f8fc7f
I20260927 14:55:31.030339 19298 tablet_service.cc:1514] Processing DeleteTablet for tablet 4305db92f72a412eb1943067f0f8fc7f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 14:55:31 UTC) from {username='slave'} at 127.0.0.1:47568
I20260927 14:55:31.030352 19167 tablet_service.cc:1514] Processing DeleteTablet for tablet 4305db92f72a412eb1943067f0f8fc7f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 14:55:31 UTC) from {username='slave'} at 127.0.0.1:34120
I20260927 14:55:31.030572 19429 tablet_service.cc:1514] Processing DeleteTablet for tablet 4305db92f72a412eb1943067f0f8fc7f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 14:55:31 UTC) from {username='slave'} at 127.0.0.1:37084
I20260927 14:55:31.030869 19535 tablet_replica.cc:330] T 4305db92f72a412eb1943067f0f8fc7f P ebb20837f7004e9ab9e294299a648287: stopping tablet replica
I20260927 14:55:31.031041 19534 tablet_replica.cc:330] T 4305db92f72a412eb1943067f0f8fc7f P 726c74bbf7d9421d9039dc2e8b32d83c: stopping tablet replica
I20260927 14:55:31.031224 19535 raft_consensus.cc:2229] T 4305db92f72a412eb1943067f0f8fc7f P ebb20837f7004e9ab9e294299a648287 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 14:55:31.031569 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6936m5/build/debug/bin/kudu with pid 19105
I20260927 14:55:31.033208 19534 raft_consensus.cc:2229] T 4305db92f72a412eb1943067f0f8fc7f P 726c74bbf7d9421d9039dc2e8b32d83c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 14:55:31.033624 19534 raft_consensus.cc:2258] T 4305db92f72a412eb1943067f0f8fc7f P 726c74bbf7d9421d9039dc2e8b32d83c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 14:55:31.034509 19536 tablet_replica.cc:330] T 4305db92f72a412eb1943067f0f8fc7f P da6f360f32fe4154b7d5282f8b627f48: stopping tablet replica
I20260927 14:55:31.034825 19536 raft_consensus.cc:2229] T 4305db92f72a412eb1943067f0f8fc7f P da6f360f32fe4154b7d5282f8b627f48 [term 1 LEADER]: Raft consensus shutting down.
I20260927 14:55:31.035229 19536 raft_consensus.cc:2258] T 4305db92f72a412eb1943067f0f8fc7f P da6f360f32fe4154b7d5282f8b627f48 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 14:55:31.035578 19534 ts_tablet_manager.cc:1905] T 4305db92f72a412eb1943067f0f8fc7f P 726c74bbf7d9421d9039dc2e8b32d83c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 14:55:31.036734 19536 ts_tablet_manager.cc:1905] T 4305db92f72a412eb1943067f0f8fc7f P da6f360f32fe4154b7d5282f8b627f48: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 14:55:31.039399 19536 ts_tablet_manager.cc:1918] T 4305db92f72a412eb1943067f0f8fc7f P da6f360f32fe4154b7d5282f8b627f48: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 14:55:31.039494 19536 log.cc:1196] T 4305db92f72a412eb1943067f0f8fc7f P da6f360f32fe4154b7d5282f8b627f48: Deleting WAL directory at /tmp/dist-test-task6936m5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790520684602251-392-0/minicluster-data/ts-2/wal/wals/4305db92f72a412eb1943067f0f8fc7f
W20260927 14:55:31.039510 19029 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:44413 (error 108)
I20260927 14:55:31.039629 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6936m5/build/debug/bin/kudu with pid 19236
I20260927 14:55:31.040457 19536 ts_tablet_manager.cc:1939] T 4305db92f72a412eb1943067f0f8fc7f P da6f360f32fe4154b7d5282f8b627f48: Deleting consensus metadata
I20260927 14:55:31.041522 19506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:55:31.041116 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44413 (local address 127.0.0.1:34126)
0927 14:55:31.041326 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0927 14:55:31.041403 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:44413: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113}
I20260927 14:55:31.041706 19030 catalog_manager.cc:4841] TS da6f360f32fe4154b7d5282f8b627f48 (127.0.98.3:41091): tablet 4305db92f72a412eb1943067f0f8fc7f (table test-table [id=c54e0e7604bd4bccacb9a82006024334]) successfully deleted
W20260927 14:55:31.041800 19029 catalog_manager.cc:4584] TS ebb20837f7004e9ab9e294299a648287 (127.0.98.1:44413): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 4305db92f72a412eb1943067f0f8fc7f: Network error: Client connection negotiation failed: client connection to 127.0.98.1:44413: connect: Connection refused (error 111)
I20260927 14:55:31.045827 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6936m5/build/debug/bin/kudu with pid 19367
I20260927 14:55:31.046054 19506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:55:31.045791 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:41977 (local address 127.0.0.1:47570)
0927 14:55:31.045908 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0927 14:55:31.045965 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:41977: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260927 14:55:31.050750 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6936m5/build/debug/bin/kudu with pid 19014
2026-09-27T14:55:31Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (994 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-task6936m5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790520684602251-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 14:55:31 dist-test-slave-xk1g krb5kdc[19540](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 14:55:31 dist-test-slave-xk1g krb5kdc[19540](info): set up 2 sockets
Sep 27 14:55:31 dist-test-slave-xk1g krb5kdc[19540](info): commencing operation
krb5kdc: starting...
W20260927 14:55:32.089144 392 mini_kdc.cc:121] Time spent starting KDC: real 1.018s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6936m5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790520684602251-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6936m5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790520684602251-392-0/krb5kdc/test-user.keytab.
Sep 27 14:55:32 dist-test-slave-xk1g krb5kdc[19540](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790520932, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T14:55:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T14:55:32Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6936m5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790520684602251-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6936m5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790520684602251-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 14:55:32.210247 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6936m5/build/debug/bin/kudu
/tmp/dist-test-task6936m5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790520684602251-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790520684602251-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790520684602251-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790520684602251-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43527
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790520684602251-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36639
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:43527
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790520684602251-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790520684602251-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790520684602251-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 14:55:32.316756 19553 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 14:55:32.317008 19553 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 14:55:32.317047 19553 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 14:55:32.321034 19553 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 14:55:32.321102 19553 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 14:55:32.321121 19553 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 14:55:32.321137 19553 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 14:55:32.321152 19553 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 14:55:32.326838 19553 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36639
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790520684602251-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790520684602251-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:43527
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790520684602251-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43527
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790520684602251-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19553
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790520684602251-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:15:17 UTC on ccycloud.quasar-ledxfe.root.comops.site
build id 4676134
I20260927 14:55:32.327804 19553 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 14:55:32.328861 19553 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 14:55:32.336884 19558 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 14:55:32.336895 19559 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 14:55:32.336918 19553 server_base.cc:1034] running on GCE node
W20260927 14:55:32.336874 19561 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 14:55:32.337653 19553 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 14:55:32.338727 19553 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 14:55:32.339905 19553 hybrid_clock.cc:648] HybridClock initialized: now 1790520932339879 us; error 39 us; skew 500 ppm
Sep 27 14:55:32 dist-test-slave-xk1g krb5kdc[19540](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790520932, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 14:55:32.342618 19553 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 14:55:32.343428 19553 webserver.cc:460] Webserver started at http://127.0.98.62:39087/ using document root <none> and password file <none>
I20260927 14:55:32.343966 19553 fs_manager.cc:359] Metadata directory not provided
I20260927 14:55:32.344050 19553 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 14:55:32.344266 19553 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 14:55:32.346254 19553 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6936m5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790520684602251-392-0/minicluster-data/master-0/data/instance:
uuid: "b466bd79a44b494592e4140206ed81c6"
format_stamp: "Formatted at 2026-09-27 14:55:32 on dist-test-slave-xk1g"
server_key: "b65d73df264f52cc944dc6c5c4450d77"
server_key_iv: "0b49c2e445c77bdfe77f0f781cd33fc4"
server_key_version: "encryptionkey@0"
I20260927 14:55:32.346798 19553 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6936m5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790520684602251-392-0/minicluster-data/master-0/wal/instance:
uuid: "b466bd79a44b494592e4140206ed81c6"
format_stamp: "Formatted at 2026-09-27 14:55:32 on dist-test-slave-xk1g"
server_key: "b65d73df264f52cc944dc6c5c4450d77"
server_key_iv: "0b49c2e445c77bdfe77f0f781cd33fc4"
server_key_version: "encryptionkey@0"
I20260927 14:55:32.350268 19553 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 14:55:32.352490 19568 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 14:55:32.353621 19553 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 14:55:32.353755 19553 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790520684602251-392-0/minicluster-data/master-0/data,/tmp/dist-test-task6936m5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790520684602251-392-0/minicluster-data/master-0/wal
uuid: "b466bd79a44b494592e4140206ed81c6"
format_stamp: "Formatted at 2026-09-27 14:55:32 on dist-test-slave-xk1g"
server_key: "b65d73df264f52cc944dc6c5c4450d77"
server_key_iv: "0b49c2e445c77bdfe77f0f781cd33fc4"
server_key_version: "encryptionkey@0"
I20260927 14:55:32.353860 19553 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790520684602251-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790520684602251-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790520684602251-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 14:55:32.370568 19553 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 14:55:32.373622 19553 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 14:55:32.373816 19553 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 14:55:32.380560 19553 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:43527
I20260927 14:55:32.380586 19620 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:43527 every 8 connection(s)
I20260927 14:55:32.381510 19553 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6936m5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790520684602251-392-0/minicluster-data/master-0/data/info.pb
I20260927 14:55:32.384136 19621 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 14:55:32.385849 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6936m5/build/debug/bin/kudu as pid 19553
I20260927 14:55:32.386004 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6936m5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790520684602251-392-0/minicluster-data/master-0/wal/instance
I20260927 14:55:32.386278 392 external_mini_cluster.cc:1428] Setting key 9c7759f50c6578e6be67ecefee6f275d
I20260927 14:55:32.390318 19621 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b466bd79a44b494592e4140206ed81c6: Bootstrap starting.
I20260927 14:55:32.392472 19621 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b466bd79a44b494592e4140206ed81c6: Neither blocks nor log segments found. Creating new log.
I20260927 14:55:32.393235 19621 log.cc:824] T 00000000000000000000000000000000 P b466bd79a44b494592e4140206ed81c6: Log is configured to *not* fsync() on all Append() calls
Sep 27 14:55:32 dist-test-slave-xk1g krb5kdc[19540](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790520932, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 14:55:32.395241 19621 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b466bd79a44b494592e4140206ed81c6: No bootstrap required, opened a new log
I20260927 14:55:32.398141 19621 raft_consensus.cc:348] T 00000000000000000000000000000000 P b466bd79a44b494592e4140206ed81c6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b466bd79a44b494592e4140206ed81c6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43527 } }
I20260927 14:55:32.398356 19621 raft_consensus.cc:374] T 00000000000000000000000000000000 P b466bd79a44b494592e4140206ed81c6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 14:55:32.398440 19621 raft_consensus.cc:729] T 00000000000000000000000000000000 P b466bd79a44b494592e4140206ed81c6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b466bd79a44b494592e4140206ed81c6, State: Initialized, Role: FOLLOWER
I20260927 14:55:32.398921 19621 consensus_queue.cc:260] T 00000000000000000000000000000000 P b466bd79a44b494592e4140206ed81c6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b466bd79a44b494592e4140206ed81c6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43527 } }
I20260927 14:55:32.399116 19621 raft_consensus.cc:388] T 00000000000000000000000000000000 P b466bd79a44b494592e4140206ed81c6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 14:55:32.399183 19621 raft_consensus.cc:482] T 00000000000000000000000000000000 P b466bd79a44b494592e4140206ed81c6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 14:55:32.399283 19621 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b466bd79a44b494592e4140206ed81c6 [term 0 FOLLOWER]: Advancing to term 1
I20260927 14:55:32.400393 19621 raft_consensus.cc:504] T 00000000000000000000000000000000 P b466bd79a44b494592e4140206ed81c6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b466bd79a44b494592e4140206ed81c6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43527 } }
I20260927 14:55:32.400734 19621 leader_election.cc:302] T 00000000000000000000000000000000 P b466bd79a44b494592e4140206ed81c6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b466bd79a44b494592e4140206ed81c6; no voters:
I20260927 14:55:32.401041 19621 leader_election.cc:288] T 00000000000000000000000000000000 P b466bd79a44b494592e4140206ed81c6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 14:55:32.401165 19626 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b466bd79a44b494592e4140206ed81c6 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 14:55:32.401261 19624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:55:32.387640 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53982 (local address 127.0.98.62:43527)
0927 14:55:32.388073 (+ 433us) server_negotiation.cc:206] Beginning negotiation
0927 14:55:32.388088 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 14:55:32.388124 (+ 36us) server_negotiation.cc:407] Connection header received
0927 14:55:32.388756 (+ 632us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:55:32.388773 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:55:32.389184 (+ 411us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:55:32.389511 (+ 327us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:55:32.390292 (+ 781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:55:32.391272 (+ 980us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:55:32.392127 (+ 855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:55:32.392642 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:55:32.392815 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:55:32.395485 (+ 2670us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:55:32.395490 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:55:32.395503 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 14:55:32.395530 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:55:32.397954 (+ 2424us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:55:32.398456 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:55:32.398463 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:55:32.398469 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:55:32.398613 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:55:32.398927 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:55:32.398933 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:55:32.398937 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:55:32.399486 (+ 549us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:55:32.399690 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0927 14:55:32.399937 (+ 247us) server_negotiation.cc:299] Negotiation successful
0927 14:55:32.400240 (+ 303us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":294,"thread_start_us":143,"threads_started":1}
I20260927 14:55:32.401410 19626 raft_consensus.cc:686] T 00000000000000000000000000000000 P b466bd79a44b494592e4140206ed81c6 [term 1 LEADER]: Becoming Leader. State: Replica: b466bd79a44b494592e4140206ed81c6, State: Running, Role: LEADER
I20260927 14:55:32.401731 19626 consensus_queue.cc:237] T 00000000000000000000000000000000 P b466bd79a44b494592e4140206ed81c6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b466bd79a44b494592e4140206ed81c6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43527 } }
I20260927 14:55:32.402261 19621 sys_catalog.cc:564] T 00000000000000000000000000000000 P b466bd79a44b494592e4140206ed81c6 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 14:55:32.404254 19627 sys_catalog.cc:455] T 00000000000000000000000000000000 P b466bd79a44b494592e4140206ed81c6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b466bd79a44b494592e4140206ed81c6. Latest consensus state: current_term: 1 leader_uuid: "b466bd79a44b494592e4140206ed81c6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b466bd79a44b494592e4140206ed81c6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43527 } } }
I20260927 14:55:32.404273 19628 sys_catalog.cc:455] T 00000000000000000000000000000000 P b466bd79a44b494592e4140206ed81c6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b466bd79a44b494592e4140206ed81c6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b466bd79a44b494592e4140206ed81c6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43527 } } }
I20260927 14:55:32.404729 19627 sys_catalog.cc:458] T 00000000000000000000000000000000 P b466bd79a44b494592e4140206ed81c6 [sys.catalog]: This master's current role is: LEADER
I20260927 14:55:32.404737 19628 sys_catalog.cc:458] T 00000000000000000000000000000000 P b466bd79a44b494592e4140206ed81c6 [sys.catalog]: This master's current role is: LEADER
W20260927 14:55:32.406780 19640 catalog_manager.cc:1558] T 00000000000000000000000000000000 P b466bd79a44b494592e4140206ed81c6: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 14:55:32.406869 19640 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 14:55:32.406951 19642 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 14:55:32.407804 19642 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 14:55:32.413496 19642 catalog_manager.cc:1347] Generated new cluster ID: e7418e28943e4612a7b8cf3f6532e9e8
I20260927 14:55:32.413563 19642 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 14:55:32.441905 19642 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 14:55:32.442968 19642 catalog_manager.cc:1504] Loading token signing keys...
I20260927 14:55:32.448040 19642 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b466bd79a44b494592e4140206ed81c6: Generated new TSK 0
I20260927 14:55:32.448761 19642 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6936m5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790520684602251-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6936m5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790520684602251-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260927 14:55:32.490191 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6936m5/build/debug/bin/kudu
/tmp/dist-test-task6936m5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790520684602251-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790520684602251-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790520684602251-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790520684602251-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43527
--keytab_file=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790520684602251-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36639
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790520684602251-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790520684602251-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790520684602251-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 14:55:32.623698 19647 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 14:55:32.623935 19647 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 14:55:32.623975 19647 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 14:55:32.627952 19647 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 14:55:32.628000 19647 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 14:55:32.628053 19647 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 14:55:32.633000 19647 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36639
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790520684602251-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790520684602251-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790520684602251-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790520684602251-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43527
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19647
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790520684602251-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:15:17 UTC on ccycloud.quasar-ledxfe.root.comops.site
build id 4676134
I20260927 14:55:32.633870 19647 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 14:55:32.634778 19647 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 14:55:32.643802 19655 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 14:55:32.643818 19653 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 14:55:32.643818 19652 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 14:55:32.644259 19647 server_base.cc:1034] running on GCE node
I20260927 14:55:32.644654 19647 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 14:55:32.645308 19647 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 14:55:32.646490 19647 hybrid_clock.cc:648] HybridClock initialized: now 1790520932646478 us; error 23 us; skew 500 ppm
Sep 27 14:55:32 dist-test-slave-xk1g krb5kdc[19540](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790520932, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 14:55:32.649137 19647 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 14:55:32.649932 19647 webserver.cc:460] Webserver started at http://127.0.98.1:43491/ using document root <none> and password file <none>
I20260927 14:55:32.650476 19647 fs_manager.cc:359] Metadata directory not provided
I20260927 14:55:32.650566 19647 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 14:55:32.650799 19647 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 14:55:32.652747 19647 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6936m5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790520684602251-392-0/minicluster-data/ts-0/data/instance:
uuid: "79667d75809c4babad831658dee965a6"
format_stamp: "Formatted at 2026-09-27 14:55:32 on dist-test-slave-xk1g"
server_key: "600008ee4bb52f4e6e75c4fee5ba3385"
server_key_iv: "99276f245e4d17d0ea4202797b526031"
server_key_version: "encryptionkey@0"
I20260927 14:55:32.653319 19647 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6936m5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790520684602251-392-0/minicluster-data/ts-0/wal/instance:
uuid: "79667d75809c4babad831658dee965a6"
format_stamp: "Formatted at 2026-09-27 14:55:32 on dist-test-slave-xk1g"
server_key: "600008ee4bb52f4e6e75c4fee5ba3385"
server_key_iv: "99276f245e4d17d0ea4202797b526031"
server_key_version: "encryptionkey@0"
I20260927 14:55:32.656813 19647 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260927 14:55:32.659304 19662 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 14:55:32.660434 19647 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260927 14:55:32.660558 19647 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790520684602251-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task6936m5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790520684602251-392-0/minicluster-data/ts-0/wal
uuid: "79667d75809c4babad831658dee965a6"
format_stamp: "Formatted at 2026-09-27 14:55:32 on dist-test-slave-xk1g"
server_key: "600008ee4bb52f4e6e75c4fee5ba3385"
server_key_iv: "99276f245e4d17d0ea4202797b526031"
server_key_version: "encryptionkey@0"
I20260927 14:55:32.660661 19647 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790520684602251-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790520684602251-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790520684602251-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 14:55:32.683495 19647 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 14:55:32.686457 19647 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 14:55:32.686666 19647 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 14:55:32.687260 19647 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 14:55:32.688084 19647 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 14:55:32.688153 19647 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 14:55:32.688222 19647 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 14:55:32.688271 19647 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 14:55:32.697268 19647 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:39441
I20260927 14:55:32.697533 19775 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:39441 every 8 connection(s)
I20260927 14:55:32.698243 19647 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6936m5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790520684602251-392-0/minicluster-data/ts-0/data/info.pb
I20260927 14:55:32.707259 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6936m5/build/debug/bin/kudu as pid 19647
I20260927 14:55:32.707420 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6936m5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790520684602251-392-0/minicluster-data/ts-0/wal/instance
Sep 27 14:55:32 dist-test-slave-xk1g krb5kdc[19540](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790520932, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 14:55:32.707757 392 external_mini_cluster.cc:1428] Setting key 4a2a22c4619f0564445feed4cf9019af
I20260927 14:55:32.712339 19624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:55:32.700057 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:56181 (local address 127.0.98.62:43527)
0927 14:55:32.700199 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0927 14:55:32.700202 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 14:55:32.701031 (+ 829us) server_negotiation.cc:407] Connection header received
0927 14:55:32.702045 (+ 1014us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:55:32.702048 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:55:32.702105 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:55:32.702209 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:55:32.703515 (+ 1306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:55:32.704112 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:55:32.705331 (+ 1219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:55:32.705563 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:55:32.705673 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:55:32.708310 (+ 2637us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:55:32.708314 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:55:32.708317 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 14:55:32.708349 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:55:32.709838 (+ 1489us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:55:32.710514 (+ 676us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:55:32.710518 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:55:32.710519 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:55:32.710597 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:55:32.711039 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:55:32.711044 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:55:32.711047 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:55:32.711320 (+ 273us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:55:32.711440 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0927 14:55:32.711950 (+ 510us) server_negotiation.cc:299] Negotiation successful
0927 14:55:32.712096 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260927 14:55:32.713099 19778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:55:32.700428 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43527 (local address 127.0.98.1:56181)
0927 14:55:32.700878 (+ 450us) negotiation.cc:107] Waiting for socket to connect
0927 14:55:32.700920 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0927 14:55:32.701845 (+ 925us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:55:32.702391 (+ 546us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:55:32.702398 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:55:32.702868 (+ 470us) client_negotiation.cc:189] Negotiated authn=SASL
0927 14:55:32.703366 (+ 498us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:55:32.703379 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:55:32.704236 (+ 857us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:55:32.704240 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:55:32.705203 (+ 963us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:55:32.705212 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:55:32.705714 (+ 502us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:55:32.705718 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:55:32.705889 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:55:32.706407 (+ 518us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 14:55:32.706422 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 14:55:32.708115 (+ 1693us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 14:55:32.709966 (+ 1851us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 14:55:32.709976 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 14:55:32.709997 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0927 14:55:32.710402 (+ 405us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 14:55:32.710692 (+ 290us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 14:55:32.710698 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 14:55:32.710703 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 14:55:32.710911 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 14:55:32.711438 (+ 527us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 14:55:32.711443 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 14:55:32.711693 (+ 250us) client_negotiation.cc:769] Sending connection context
0927 14:55:32.711857 (+ 164us) client_negotiation.cc:240] Negotiation successful
0927 14:55:32.712130 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":266,"thread_start_us":117,"threads_started":1}
I20260927 14:55:32.714344 19776 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43527
I20260927 14:55:32.714573 19776 heartbeater.cc:464] Registering TS with master...
I20260927 14:55:32.715071 19776 heartbeater.cc:511] Master 127.0.98.62:43527 requested a full tablet report, sending...
I20260927 14:55:32.716686 19585 ts_manager.cc:194] Registered new tserver with Master: 79667d75809c4babad831658dee965a6 (127.0.98.1:39441)
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260927 14:55:32.717922 19585 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:56181
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6936m5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790520684602251-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6936m5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790520684602251-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260927 14:55:32.736462 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6936m5/build/debug/bin/kudu
/tmp/dist-test-task6936m5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790520684602251-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790520684602251-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790520684602251-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790520684602251-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43527
--keytab_file=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790520684602251-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36639
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790520684602251-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790520684602251-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790520684602251-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 14:55:32.845993 19781 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 14:55:32.846223 19781 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 14:55:32.846263 19781 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 14:55:32.850220 19781 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 14:55:32.850278 19781 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 14:55:32.850363 19781 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 14:55:32.855737 19781 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36639
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790520684602251-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790520684602251-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790520684602251-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790520684602251-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43527
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19781
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790520684602251-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:15:17 UTC on ccycloud.quasar-ledxfe.root.comops.site
build id 4676134
I20260927 14:55:32.856775 19781 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 14:55:32.857882 19781 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 14:55:32.867503 19789 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 14:55:32.867491 19787 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 14:55:32.867491 19786 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 14:55:32.867588 19781 server_base.cc:1034] running on GCE node
I20260927 14:55:32.868151 19781 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 14:55:32.868767 19781 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 14:55:32.869921 19781 hybrid_clock.cc:648] HybridClock initialized: now 1790520932869889 us; error 46 us; skew 500 ppm
Sep 27 14:55:32 dist-test-slave-xk1g krb5kdc[19540](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790520932, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 14:55:32.872649 19781 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 14:55:32.873518 19781 webserver.cc:460] Webserver started at http://127.0.98.2:40109/ using document root <none> and password file <none>
I20260927 14:55:32.874064 19781 fs_manager.cc:359] Metadata directory not provided
I20260927 14:55:32.874123 19781 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 14:55:32.874336 19781 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 14:55:32.876302 19781 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6936m5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790520684602251-392-0/minicluster-data/ts-1/data/instance:
uuid: "681c1b7e05f544be8e7a97864d9bd98d"
format_stamp: "Formatted at 2026-09-27 14:55:32 on dist-test-slave-xk1g"
server_key: "94783ff5056ae3a48a0e86e9c5e4a49c"
server_key_iv: "d056cad0a3ac8c25dd14382c6aa1363c"
server_key_version: "encryptionkey@0"
I20260927 14:55:32.876849 19781 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6936m5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790520684602251-392-0/minicluster-data/ts-1/wal/instance:
uuid: "681c1b7e05f544be8e7a97864d9bd98d"
format_stamp: "Formatted at 2026-09-27 14:55:32 on dist-test-slave-xk1g"
server_key: "94783ff5056ae3a48a0e86e9c5e4a49c"
server_key_iv: "d056cad0a3ac8c25dd14382c6aa1363c"
server_key_version: "encryptionkey@0"
I20260927 14:55:32.880467 19781 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260927 14:55:32.882988 19796 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 14:55:32.884169 19781 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 14:55:32.884297 19781 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790520684602251-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task6936m5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790520684602251-392-0/minicluster-data/ts-1/wal
uuid: "681c1b7e05f544be8e7a97864d9bd98d"
format_stamp: "Formatted at 2026-09-27 14:55:32 on dist-test-slave-xk1g"
server_key: "94783ff5056ae3a48a0e86e9c5e4a49c"
server_key_iv: "d056cad0a3ac8c25dd14382c6aa1363c"
server_key_version: "encryptionkey@0"
I20260927 14:55:32.884398 19781 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790520684602251-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790520684602251-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790520684602251-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 14:55:32.905620 19781 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 14:55:32.908422 19781 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 14:55:32.908591 19781 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 14:55:32.909154 19781 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 14:55:32.909901 19781 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 14:55:32.909945 19781 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 14:55:32.909983 19781 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 14:55:32.910004 19781 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 14:55:32.918356 19781 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:36349
I20260927 14:55:32.918430 19909 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:36349 every 8 connection(s)
I20260927 14:55:32.919229 19781 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6936m5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790520684602251-392-0/minicluster-data/ts-1/data/info.pb
I20260927 14:55:32.922711 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6936m5/build/debug/bin/kudu as pid 19781
I20260927 14:55:32.922842 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6936m5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790520684602251-392-0/minicluster-data/ts-1/wal/instance
I20260927 14:55:32.923185 392 external_mini_cluster.cc:1428] Setting key be5215df2f40c98ea024acc3efce8eb6
Sep 27 14:55:32 dist-test-slave-xk1g krb5kdc[19540](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790520932, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260927 14:55:32.933337 19624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:55:32.920973 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:32793 (local address 127.0.98.62:43527)
0927 14:55:32.921137 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0927 14:55:32.921141 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 14:55:32.921977 (+ 836us) server_negotiation.cc:407] Connection header received
0927 14:55:32.923049 (+ 1072us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:55:32.923052 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:55:32.923101 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:55:32.923188 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:55:32.924618 (+ 1430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:55:32.925211 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:55:32.926708 (+ 1497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:55:32.927090 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:55:32.927225 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:55:32.929652 (+ 2427us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:55:32.929657 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:55:32.929662 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 14:55:32.929695 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:55:32.931343 (+ 1648us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:55:32.931821 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:55:32.931826 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:55:32.931828 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:55:32.931919 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:55:32.932236 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:55:32.932239 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:55:32.932242 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:55:32.932460 (+ 218us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:55:32.932561 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0927 14:55:32.933105 (+ 544us) server_negotiation.cc:299] Negotiation successful
0927 14:55:32.933220 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260927 14:55:32.934211 19912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:55:32.921401 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43527 (local address 127.0.98.2:32793)
0927 14:55:32.921828 (+ 427us) negotiation.cc:107] Waiting for socket to connect
0927 14:55:32.921868 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0927 14:55:32.922755 (+ 887us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:55:32.923394 (+ 639us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:55:32.923403 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:55:32.923872 (+ 469us) client_negotiation.cc:189] Negotiated authn=SASL
0927 14:55:32.924472 (+ 600us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:55:32.924514 (+ 42us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:55:32.925342 (+ 828us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:55:32.925348 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:55:32.926585 (+ 1237us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:55:32.926594 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:55:32.927238 (+ 644us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:55:32.927241 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:55:32.927382 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:55:32.927982 (+ 600us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 14:55:32.927998 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 14:55:32.929485 (+ 1487us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 14:55:32.931465 (+ 1980us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 14:55:32.931470 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 14:55:32.931480 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0927 14:55:32.931718 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 14:55:32.932017 (+ 299us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 14:55:32.932019 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 14:55:32.932021 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 14:55:32.932143 (+ 122us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 14:55:32.932559 (+ 416us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 14:55:32.932564 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 14:55:32.932838 (+ 274us) client_negotiation.cc:769] Sending connection context
0927 14:55:32.932998 (+ 160us) client_negotiation.cc:240] Negotiation successful
0927 14:55:32.933311 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":254,"thread_start_us":111,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260927 14:55:32.935453 19910 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43527
I20260927 14:55:32.935724 19910 heartbeater.cc:464] Registering TS with master...
I20260927 14:55:32.936326 19910 heartbeater.cc:511] Master 127.0.98.62:43527 requested a full tablet report, sending...
I20260927 14:55:32.937494 19585 ts_manager.cc:194] Registered new tserver with Master: 681c1b7e05f544be8e7a97864d9bd98d (127.0.98.2:36349)
I20260927 14:55:32.938354 19585 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:32793
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6936m5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790520684602251-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6936m5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790520684602251-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260927 14:55:32.952230 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6936m5/build/debug/bin/kudu
/tmp/dist-test-task6936m5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790520684602251-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790520684602251-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790520684602251-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790520684602251-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43527
--keytab_file=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790520684602251-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36639
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790520684602251-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790520684602251-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790520684602251-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 14:55:33.061360 19915 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 14:55:33.061595 19915 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 14:55:33.061664 19915 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 14:55:33.065950 19915 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 14:55:33.066000 19915 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 14:55:33.066083 19915 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 14:55:33.071228 19915 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36639
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790520684602251-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790520684602251-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790520684602251-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790520684602251-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43527
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19915
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790520684602251-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:15:17 UTC on ccycloud.quasar-ledxfe.root.comops.site
build id 4676134
I20260927 14:55:33.072268 19915 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 14:55:33.073276 19915 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 14:55:33.082448 19923 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 14:55:33.082472 19921 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 14:55:33.082455 19920 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 14:55:33.082629 19915 server_base.cc:1034] running on GCE node
I20260927 14:55:33.083137 19915 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 14:55:33.083745 19915 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 14:55:33.084909 19915 hybrid_clock.cc:648] HybridClock initialized: now 1790520933084890 us; error 30 us; skew 500 ppm
Sep 27 14:55:33 dist-test-slave-xk1g krb5kdc[19540](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790520933, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 14:55:33.087703 19915 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 14:55:33.088557 19915 webserver.cc:460] Webserver started at http://127.0.98.3:42871/ using document root <none> and password file <none>
I20260927 14:55:33.089118 19915 fs_manager.cc:359] Metadata directory not provided
I20260927 14:55:33.089181 19915 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 14:55:33.089354 19915 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 14:55:33.091241 19915 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6936m5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790520684602251-392-0/minicluster-data/ts-2/data/instance:
uuid: "e11561118c2445f29637727c10fbde4a"
format_stamp: "Formatted at 2026-09-27 14:55:33 on dist-test-slave-xk1g"
server_key: "af075622af51ac5db2d24449ea4cdc7a"
server_key_iv: "3f375521f99dd06464eedc7d4d7ca54b"
server_key_version: "encryptionkey@0"
I20260927 14:55:33.091744 19915 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6936m5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790520684602251-392-0/minicluster-data/ts-2/wal/instance:
uuid: "e11561118c2445f29637727c10fbde4a"
format_stamp: "Formatted at 2026-09-27 14:55:33 on dist-test-slave-xk1g"
server_key: "af075622af51ac5db2d24449ea4cdc7a"
server_key_iv: "3f375521f99dd06464eedc7d4d7ca54b"
server_key_version: "encryptionkey@0"
I20260927 14:55:33.095304 19915 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260927 14:55:33.097575 19930 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 14:55:33.098773 19915 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 14:55:33.098917 19915 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790520684602251-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task6936m5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790520684602251-392-0/minicluster-data/ts-2/wal
uuid: "e11561118c2445f29637727c10fbde4a"
format_stamp: "Formatted at 2026-09-27 14:55:33 on dist-test-slave-xk1g"
server_key: "af075622af51ac5db2d24449ea4cdc7a"
server_key_iv: "3f375521f99dd06464eedc7d4d7ca54b"
server_key_version: "encryptionkey@0"
I20260927 14:55:33.099025 19915 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790520684602251-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790520684602251-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790520684602251-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 14:55:33.112665 19915 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 14:55:33.116007 19915 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 14:55:33.116215 19915 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 14:55:33.116889 19915 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 14:55:33.117770 19915 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 14:55:33.117821 19915 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 14:55:33.117861 19915 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 14:55:33.117882 19915 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 14:55:33.126678 19915 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:46011
I20260927 14:55:33.126713 20043 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:46011 every 8 connection(s)
I20260927 14:55:33.127586 19915 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6936m5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790520684602251-392-0/minicluster-data/ts-2/data/info.pb
I20260927 14:55:33.128072 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6936m5/build/debug/bin/kudu as pid 19915
I20260927 14:55:33.128216 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6936m5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790520684602251-392-0/minicluster-data/ts-2/wal/instance
I20260927 14:55:33.128523 392 external_mini_cluster.cc:1428] Setting key 852d7c08857b867798f86e63c066f650
Sep 27 14:55:33 dist-test-slave-xk1g krb5kdc[19540](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790520933, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 14:55:33.140269 19624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:55:33.129624 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:60391 (local address 127.0.98.62:43527)
0927 14:55:33.129756 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0927 14:55:33.129761 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 14:55:33.130711 (+ 950us) server_negotiation.cc:407] Connection header received
0927 14:55:33.131533 (+ 822us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:55:33.131536 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:55:33.131606 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:55:33.131698 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:55:33.133005 (+ 1307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:55:33.133573 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:55:33.134637 (+ 1064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:55:33.134847 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:55:33.134923 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:55:33.137158 (+ 2235us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:55:33.137161 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:55:33.137163 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 14:55:33.137194 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:55:33.138615 (+ 1421us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:55:33.139037 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:55:33.139039 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:55:33.139041 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:55:33.139108 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:55:33.139392 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:55:33.139393 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:55:33.139394 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:55:33.139548 (+ 154us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:55:33.139619 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0927 14:55:33.140045 (+ 426us) server_negotiation.cc:299] Negotiation successful
0927 14:55:33.140143 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260927 14:55:33.141050 20046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:55:33.130125 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43527 (local address 127.0.98.3:60391)
0927 14:55:33.130558 (+ 433us) negotiation.cc:107] Waiting for socket to connect
0927 14:55:33.130599 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0927 14:55:33.131322 (+ 723us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:55:33.131911 (+ 589us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:55:33.131919 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:55:33.132403 (+ 484us) client_negotiation.cc:189] Negotiated authn=SASL
0927 14:55:33.132879 (+ 476us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:55:33.132893 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:55:33.133682 (+ 789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:55:33.133685 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:55:33.134541 (+ 856us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:55:33.134549 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:55:33.134965 (+ 416us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:55:33.134968 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:55:33.135108 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:55:33.135621 (+ 513us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 14:55:33.135636 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 14:55:33.136974 (+ 1338us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 14:55:33.138709 (+ 1735us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 14:55:33.138714 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 14:55:33.138725 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0927 14:55:33.138958 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 14:55:33.139182 (+ 224us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 14:55:33.139184 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 14:55:33.139185 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 14:55:33.139319 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 14:55:33.139630 (+ 311us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 14:55:33.139635 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 14:55:33.139854 (+ 219us) client_negotiation.cc:769] Sending connection context
0927 14:55:33.140002 (+ 148us) client_negotiation.cc:240] Negotiation successful
0927 14:55:33.140237 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":255,"thread_start_us":107,"threads_started":1}
I20260927 14:55:33.142201 20044 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43527
I20260927 14:55:33.142445 20044 heartbeater.cc:464] Registering TS with master...
I20260927 14:55:33.142994 20044 heartbeater.cc:511] Master 127.0.98.62:43527 requested a full tablet report, sending...
I20260927 14:55:33.144147 19584 ts_manager.cc:194] Registered new tserver with Master: e11561118c2445f29637727c10fbde4a (127.0.98.3:46011)
I20260927 14:55:33.144769 19584 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:60391
I20260927 14:55:33.156231 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 14:55:33.165825 19624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:55:33.158502 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53988 (local address 127.0.98.62:43527)
0927 14:55:33.158692 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0927 14:55:33.158698 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 14:55:33.158890 (+ 192us) server_negotiation.cc:407] Connection header received
0927 14:55:33.159036 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:55:33.159039 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:55:33.159093 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:55:33.159181 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:55:33.159901 (+ 720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:55:33.160423 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:55:33.161298 (+ 875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:55:33.161519 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:55:33.161634 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:55:33.162885 (+ 1251us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:55:33.162889 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:55:33.162892 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 14:55:33.162922 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:55:33.164110 (+ 1188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:55:33.164510 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:55:33.164514 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:55:33.164515 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:55:33.164605 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:55:33.164873 (+ 268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:55:33.164876 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:55:33.164878 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:55:33.165162 (+ 284us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:55:33.165302 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0927 14:55:33.165540 (+ 238us) server_negotiation.cc:299] Negotiation successful
0927 14:55:33.165660 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260927 14:55:33.168552 19584 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53988:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 14:55:33.171733 19584 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 14:55:33.185412 20055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:55:33.179950 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:46011 (local address 127.0.0.1:47840)
0927 14:55:33.180390 (+ 440us) negotiation.cc:107] Waiting for socket to connect
0927 14:55:33.180441 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0927 14:55:33.180556 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:55:33.181394 (+ 838us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:55:33.181400 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:55:33.181422 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 14:55:33.181564 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:55:33.181573 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:55:33.182869 (+ 1296us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:55:33.182874 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:55:33.183764 (+ 890us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:55:33.183772 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:55:33.184875 (+ 1103us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:55:33.184878 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:55:33.184958 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:55:33.184987 (+ 29us) client_negotiation.cc:769] Sending connection context
0927 14:55:33.185147 (+ 160us) client_negotiation.cc:240] Negotiation successful
0927 14:55:33.185271 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":263,"thread_start_us":78,"threads_started":1}
I20260927 14:55:33.185447 20057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:55:33.180083 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47840 (local address 127.0.98.3:46011)
0927 14:55:33.180874 (+ 791us) server_negotiation.cc:206] Beginning negotiation
0927 14:55:33.180889 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 14:55:33.180912 (+ 23us) server_negotiation.cc:407] Connection header received
0927 14:55:33.180993 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:55:33.181001 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:55:33.181232 (+ 231us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:55:33.181394 (+ 162us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 14:55:33.181704 (+ 310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:55:33.182743 (+ 1039us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:55:33.183901 (+ 1158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:55:33.184761 (+ 860us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:55:33.184871 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:55:33.185005 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0927 14:55:33.185201 (+ 196us) server_negotiation.cc:299] Negotiation successful
0927 14:55:33.185309 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":677,"thread_start_us":172,"threads_started":1}
I20260927 14:55:33.186982 20058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:55:33.180676 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:36349 (local address 127.0.0.1:48110)
0927 14:55:33.180954 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0927 14:55:33.180968 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 14:55:33.181031 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:55:33.182178 (+ 1147us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:55:33.182184 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:55:33.182206 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 14:55:33.182329 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:55:33.182336 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:55:33.183764 (+ 1428us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:55:33.183767 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:55:33.185817 (+ 2050us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:55:33.185829 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:55:33.186600 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:55:33.186604 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:55:33.186685 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:55:33.186706 (+ 21us) client_negotiation.cc:769] Sending connection context
0927 14:55:33.186769 (+ 63us) client_negotiation.cc:240] Negotiation successful
0927 14:55:33.186850 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":222,"mutex_wait_us":98,"thread_start_us":89,"threads_started":1}
I20260927 14:55:33.187212 20056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:55:33.180026 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:39441 (local address 127.0.0.1:56740)
0927 14:55:33.180915 (+ 889us) negotiation.cc:107] Waiting for socket to connect
0927 14:55:33.180928 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 14:55:33.181022 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:55:33.182246 (+ 1224us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:55:33.182249 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:55:33.182261 (+ 12us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 14:55:33.182405 (+ 144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:55:33.182413 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:55:33.183786 (+ 1373us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:55:33.183789 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:55:33.185086 (+ 1297us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:55:33.185096 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:55:33.186856 (+ 1760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:55:33.186860 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:55:33.186943 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:55:33.186965 (+ 22us) client_negotiation.cc:769] Sending connection context
0927 14:55:33.187011 (+ 46us) client_negotiation.cc:240] Negotiation successful
0927 14:55:33.187073 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":820,"mutex_wait_us":48,"thread_start_us":76,"threads_started":1}
I20260927 14:55:33.187696 20060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:55:33.181405 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48110 (local address 127.0.98.2:36349)
0927 14:55:33.181739 (+ 334us) server_negotiation.cc:206] Beginning negotiation
0927 14:55:33.181753 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 14:55:33.181775 (+ 22us) server_negotiation.cc:407] Connection header received
0927 14:55:33.181849 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:55:33.181853 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:55:33.182029 (+ 176us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:55:33.182186 (+ 157us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 14:55:33.182458 (+ 272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:55:33.183624 (+ 1166us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:55:33.185954 (+ 2330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:55:33.186475 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:55:33.186604 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:55:33.186745 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0927 14:55:33.186802 (+ 57us) server_negotiation.cc:299] Negotiation successful
0927 14:55:33.186886 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":215,"thread_start_us":112,"threads_started":1}
I20260927 14:55:33.187850 20059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:55:33.180916 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56740 (local address 127.0.98.1:39441)
0927 14:55:33.181855 (+ 939us) server_negotiation.cc:206] Beginning negotiation
0927 14:55:33.181868 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 14:55:33.181887 (+ 19us) server_negotiation.cc:407] Connection header received
0927 14:55:33.181946 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:55:33.181949 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:55:33.182088 (+ 139us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:55:33.182370 (+ 282us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 14:55:33.182533 (+ 163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:55:33.183641 (+ 1108us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:55:33.185244 (+ 1603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:55:33.186719 (+ 1475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:55:33.187400 (+ 681us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:55:33.187574 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0927 14:55:33.187621 (+ 47us) server_negotiation.cc:299] Negotiation successful
0927 14:55:33.187704 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":825,"thread_start_us":68,"threads_started":1}
I20260927 14:55:33.188741 19978 tablet_service.cc:1467] Processing CreateTablet for tablet a67e9674ce154bef9eb3eec2dfe65cef (DEFAULT_TABLE table=test-table [id=4985f57021f644ee8f4352aa26d0a02a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 14:55:33.189875 19978 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a67e9674ce154bef9eb3eec2dfe65cef. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 14:55:33.189994 19844 tablet_service.cc:1467] Processing CreateTablet for tablet a67e9674ce154bef9eb3eec2dfe65cef (DEFAULT_TABLE table=test-table [id=4985f57021f644ee8f4352aa26d0a02a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 14:55:33.190418 19710 tablet_service.cc:1467] Processing CreateTablet for tablet a67e9674ce154bef9eb3eec2dfe65cef (DEFAULT_TABLE table=test-table [id=4985f57021f644ee8f4352aa26d0a02a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 14:55:33.190858 19844 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a67e9674ce154bef9eb3eec2dfe65cef. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 14:55:33.191339 19710 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a67e9674ce154bef9eb3eec2dfe65cef. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 14:55:33.196415 20061 tablet_bootstrap.cc:492] T a67e9674ce154bef9eb3eec2dfe65cef P 681c1b7e05f544be8e7a97864d9bd98d: Bootstrap starting.
I20260927 14:55:33.197775 20062 tablet_bootstrap.cc:492] T a67e9674ce154bef9eb3eec2dfe65cef P e11561118c2445f29637727c10fbde4a: Bootstrap starting.
I20260927 14:55:33.198076 20063 tablet_bootstrap.cc:492] T a67e9674ce154bef9eb3eec2dfe65cef P 79667d75809c4babad831658dee965a6: Bootstrap starting.
I20260927 14:55:33.199110 20061 tablet_bootstrap.cc:654] T a67e9674ce154bef9eb3eec2dfe65cef P 681c1b7e05f544be8e7a97864d9bd98d: Neither blocks nor log segments found. Creating new log.
I20260927 14:55:33.199604 20062 tablet_bootstrap.cc:654] T a67e9674ce154bef9eb3eec2dfe65cef P e11561118c2445f29637727c10fbde4a: Neither blocks nor log segments found. Creating new log.
I20260927 14:55:33.200268 20063 tablet_bootstrap.cc:654] T a67e9674ce154bef9eb3eec2dfe65cef P 79667d75809c4babad831658dee965a6: Neither blocks nor log segments found. Creating new log.
I20260927 14:55:33.200322 20061 log.cc:824] T a67e9674ce154bef9eb3eec2dfe65cef P 681c1b7e05f544be8e7a97864d9bd98d: Log is configured to *not* fsync() on all Append() calls
I20260927 14:55:33.200590 20062 log.cc:824] T a67e9674ce154bef9eb3eec2dfe65cef P e11561118c2445f29637727c10fbde4a: Log is configured to *not* fsync() on all Append() calls
I20260927 14:55:33.201428 20063 log.cc:824] T a67e9674ce154bef9eb3eec2dfe65cef P 79667d75809c4babad831658dee965a6: Log is configured to *not* fsync() on all Append() calls
I20260927 14:55:33.202441 20061 tablet_bootstrap.cc:492] T a67e9674ce154bef9eb3eec2dfe65cef P 681c1b7e05f544be8e7a97864d9bd98d: No bootstrap required, opened a new log
I20260927 14:55:33.202441 20062 tablet_bootstrap.cc:492] T a67e9674ce154bef9eb3eec2dfe65cef P e11561118c2445f29637727c10fbde4a: No bootstrap required, opened a new log
I20260927 14:55:33.202632 20062 ts_tablet_manager.cc:1397] T a67e9674ce154bef9eb3eec2dfe65cef P e11561118c2445f29637727c10fbde4a: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260927 14:55:33.202628 20061 ts_tablet_manager.cc:1397] T a67e9674ce154bef9eb3eec2dfe65cef P 681c1b7e05f544be8e7a97864d9bd98d: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260927 14:55:33.203542 20063 tablet_bootstrap.cc:492] T a67e9674ce154bef9eb3eec2dfe65cef P 79667d75809c4babad831658dee965a6: No bootstrap required, opened a new log
I20260927 14:55:33.203936 20063 ts_tablet_manager.cc:1397] T a67e9674ce154bef9eb3eec2dfe65cef P 79667d75809c4babad831658dee965a6: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260927 14:55:33.206595 20062 raft_consensus.cc:348] T a67e9674ce154bef9eb3eec2dfe65cef P e11561118c2445f29637727c10fbde4a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e11561118c2445f29637727c10fbde4a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46011 } } peers { permanent_uuid: "79667d75809c4babad831658dee965a6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39441 } } peers { permanent_uuid: "681c1b7e05f544be8e7a97864d9bd98d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36349 } }
I20260927 14:55:33.206595 20061 raft_consensus.cc:348] T a67e9674ce154bef9eb3eec2dfe65cef P 681c1b7e05f544be8e7a97864d9bd98d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e11561118c2445f29637727c10fbde4a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46011 } } peers { permanent_uuid: "79667d75809c4babad831658dee965a6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39441 } } peers { permanent_uuid: "681c1b7e05f544be8e7a97864d9bd98d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36349 } }
I20260927 14:55:33.206879 20061 raft_consensus.cc:374] T a67e9674ce154bef9eb3eec2dfe65cef P 681c1b7e05f544be8e7a97864d9bd98d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 14:55:33.206879 20062 raft_consensus.cc:374] T a67e9674ce154bef9eb3eec2dfe65cef P e11561118c2445f29637727c10fbde4a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 14:55:33.206741 20063 raft_consensus.cc:348] T a67e9674ce154bef9eb3eec2dfe65cef P 79667d75809c4babad831658dee965a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e11561118c2445f29637727c10fbde4a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46011 } } peers { permanent_uuid: "79667d75809c4babad831658dee965a6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39441 } } peers { permanent_uuid: "681c1b7e05f544be8e7a97864d9bd98d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36349 } }
I20260927 14:55:33.206939 20061 raft_consensus.cc:729] T a67e9674ce154bef9eb3eec2dfe65cef P 681c1b7e05f544be8e7a97864d9bd98d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 681c1b7e05f544be8e7a97864d9bd98d, State: Initialized, Role: FOLLOWER
I20260927 14:55:33.206947 20062 raft_consensus.cc:729] T a67e9674ce154bef9eb3eec2dfe65cef P e11561118c2445f29637727c10fbde4a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e11561118c2445f29637727c10fbde4a, State: Initialized, Role: FOLLOWER
I20260927 14:55:33.206976 20063 raft_consensus.cc:374] T a67e9674ce154bef9eb3eec2dfe65cef P 79667d75809c4babad831658dee965a6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 14:55:33.207037 20063 raft_consensus.cc:729] T a67e9674ce154bef9eb3eec2dfe65cef P 79667d75809c4babad831658dee965a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 79667d75809c4babad831658dee965a6, State: Initialized, Role: FOLLOWER
I20260927 14:55:33.207409 20063 consensus_queue.cc:260] T a67e9674ce154bef9eb3eec2dfe65cef P 79667d75809c4babad831658dee965a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e11561118c2445f29637727c10fbde4a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46011 } } peers { permanent_uuid: "79667d75809c4babad831658dee965a6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39441 } } peers { permanent_uuid: "681c1b7e05f544be8e7a97864d9bd98d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36349 } }
I20260927 14:55:33.207415 20062 consensus_queue.cc:260] T a67e9674ce154bef9eb3eec2dfe65cef P e11561118c2445f29637727c10fbde4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e11561118c2445f29637727c10fbde4a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46011 } } peers { permanent_uuid: "79667d75809c4babad831658dee965a6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39441 } } peers { permanent_uuid: "681c1b7e05f544be8e7a97864d9bd98d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36349 } }
I20260927 14:55:33.207417 20061 consensus_queue.cc:260] T a67e9674ce154bef9eb3eec2dfe65cef P 681c1b7e05f544be8e7a97864d9bd98d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e11561118c2445f29637727c10fbde4a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46011 } } peers { permanent_uuid: "79667d75809c4babad831658dee965a6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39441 } } peers { permanent_uuid: "681c1b7e05f544be8e7a97864d9bd98d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36349 } }
I20260927 14:55:33.208057 20044 heartbeater.cc:503] Master 127.0.98.62:43527 was elected leader, sending a full tablet report...
I20260927 14:55:33.208328 19776 heartbeater.cc:503] Master 127.0.98.62:43527 was elected leader, sending a full tablet report...
I20260927 14:55:33.208371 20063 ts_tablet_manager.cc:1428] T a67e9674ce154bef9eb3eec2dfe65cef P 79667d75809c4babad831658dee965a6: Time spent starting tablet: real 0.004s user 0.000s sys 0.004s
I20260927 14:55:33.209214 20062 ts_tablet_manager.cc:1428] T a67e9674ce154bef9eb3eec2dfe65cef P e11561118c2445f29637727c10fbde4a: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260927 14:55:33.210903 20061 ts_tablet_manager.cc:1428] T a67e9674ce154bef9eb3eec2dfe65cef P 681c1b7e05f544be8e7a97864d9bd98d: Time spent starting tablet: real 0.008s user 0.006s sys 0.000s
I20260927 14:55:33.211217 19910 heartbeater.cc:503] Master 127.0.98.62:43527 was elected leader, sending a full tablet report...
I20260927 14:55:33.223747 20067 raft_consensus.cc:482] T a67e9674ce154bef9eb3eec2dfe65cef P e11561118c2445f29637727c10fbde4a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 14:55:33.223901 20067 raft_consensus.cc:504] T a67e9674ce154bef9eb3eec2dfe65cef P e11561118c2445f29637727c10fbde4a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e11561118c2445f29637727c10fbde4a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46011 } } peers { permanent_uuid: "79667d75809c4babad831658dee965a6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39441 } } peers { permanent_uuid: "681c1b7e05f544be8e7a97864d9bd98d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36349 } }
I20260927 14:55:33.224853 20067 leader_election.cc:288] T a67e9674ce154bef9eb3eec2dfe65cef P e11561118c2445f29637727c10fbde4a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 79667d75809c4babad831658dee965a6 (127.0.98.1:39441), 681c1b7e05f544be8e7a97864d9bd98d (127.0.98.2:36349)
I20260927 14:55:33.228636 20059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:55:33.225275 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:53567 (local address 127.0.98.1:39441)
0927 14:55:33.225417 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0927 14:55:33.225422 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 14:55:33.225441 (+ 19us) server_negotiation.cc:407] Connection header received
0927 14:55:33.225490 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:55:33.225494 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:55:33.225541 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:55:33.225606 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 14:55:33.225967 (+ 361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:55:33.226554 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:55:33.227631 (+ 1077us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:55:33.228065 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:55:33.228150 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:55:33.228198 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0927 14:55:33.228433 (+ 235us) server_negotiation.cc:299] Negotiation successful
0927 14:55:33.228515 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260927 14:55:33.228616 20046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:55:33.225058 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:39441 (local address 127.0.98.3:53567)
0927 14:55:33.225298 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0927 14:55:33.225315 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 14:55:33.225400 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:55:33.225646 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:55:33.225649 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:55:33.225679 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 14:55:33.225831 (+ 152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:55:33.225838 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:55:33.226681 (+ 843us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:55:33.226684 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:55:33.227500 (+ 816us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:55:33.227509 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:55:33.228189 (+ 680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:55:33.228193 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:55:33.228306 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:55:33.228328 (+ 22us) client_negotiation.cc:769] Sending connection context
0927 14:55:33.228399 (+ 71us) client_negotiation.cc:240] Negotiation successful
0927 14:55:33.228465 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":150,"spinlock_wait_cycles":19840}
I20260927 14:55:33.228739 20060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:55:33.225479 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:40715 (local address 127.0.98.2:36349)
0927 14:55:33.225605 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0927 14:55:33.225611 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 14:55:33.225726 (+ 115us) server_negotiation.cc:407] Connection header received
0927 14:55:33.225829 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:55:33.225831 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:55:33.225881 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:55:33.225956 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 14:55:33.226215 (+ 259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:55:33.226720 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:55:33.227755 (+ 1035us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:55:33.228186 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:55:33.228288 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:55:33.228345 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0927 14:55:33.228554 (+ 209us) server_negotiation.cc:299] Negotiation successful
0927 14:55:33.228623 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260927 14:55:33.228966 20070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:55:33.225370 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:36349 (local address 127.0.98.3:40715)
0927 14:55:33.225638 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0927 14:55:33.225650 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 14:55:33.225745 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:55:33.225979 (+ 234us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:55:33.225982 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:55:33.225998 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 14:55:33.226105 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:55:33.226111 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:55:33.226829 (+ 718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:55:33.226832 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:55:33.227640 (+ 808us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:55:33.227650 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:55:33.228319 (+ 669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:55:33.228321 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:55:33.228426 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:55:33.228450 (+ 24us) client_negotiation.cc:769] Sending connection context
0927 14:55:33.228525 (+ 75us) client_negotiation.cc:240] Negotiation successful
0927 14:55:33.228587 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":212,"thread_start_us":99,"threads_started":1}
I20260927 14:55:33.229185 19730 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a67e9674ce154bef9eb3eec2dfe65cef" candidate_uuid: "e11561118c2445f29637727c10fbde4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "79667d75809c4babad831658dee965a6" is_pre_election: true
I20260927 14:55:33.229413 19730 raft_consensus.cc:2454] T a67e9674ce154bef9eb3eec2dfe65cef P 79667d75809c4babad831658dee965a6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e11561118c2445f29637727c10fbde4a in term 0.
I20260927 14:55:33.229513 19864 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a67e9674ce154bef9eb3eec2dfe65cef" candidate_uuid: "e11561118c2445f29637727c10fbde4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "681c1b7e05f544be8e7a97864d9bd98d" is_pre_election: true
I20260927 14:55:33.229732 19864 raft_consensus.cc:2454] T a67e9674ce154bef9eb3eec2dfe65cef P 681c1b7e05f544be8e7a97864d9bd98d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e11561118c2445f29637727c10fbde4a in term 0.
I20260927 14:55:33.229835 19934 leader_election.cc:302] T a67e9674ce154bef9eb3eec2dfe65cef P e11561118c2445f29637727c10fbde4a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 79667d75809c4babad831658dee965a6, e11561118c2445f29637727c10fbde4a; no voters:
I20260927 14:55:33.230088 20067 raft_consensus.cc:2781] T a67e9674ce154bef9eb3eec2dfe65cef P e11561118c2445f29637727c10fbde4a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 14:55:33.230193 20067 raft_consensus.cc:482] T a67e9674ce154bef9eb3eec2dfe65cef P e11561118c2445f29637727c10fbde4a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 14:55:33.230244 20067 raft_consensus.cc:3037] T a67e9674ce154bef9eb3eec2dfe65cef P e11561118c2445f29637727c10fbde4a [term 0 FOLLOWER]: Advancing to term 1
I20260927 14:55:33.231309 20067 raft_consensus.cc:504] T a67e9674ce154bef9eb3eec2dfe65cef P e11561118c2445f29637727c10fbde4a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e11561118c2445f29637727c10fbde4a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46011 } } peers { permanent_uuid: "79667d75809c4babad831658dee965a6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39441 } } peers { permanent_uuid: "681c1b7e05f544be8e7a97864d9bd98d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36349 } }
I20260927 14:55:33.231560 20067 leader_election.cc:288] T a67e9674ce154bef9eb3eec2dfe65cef P e11561118c2445f29637727c10fbde4a [CANDIDATE]: Term 1 election: Requested vote from peers 79667d75809c4babad831658dee965a6 (127.0.98.1:39441), 681c1b7e05f544be8e7a97864d9bd98d (127.0.98.2:36349)
I20260927 14:55:33.231910 19730 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a67e9674ce154bef9eb3eec2dfe65cef" candidate_uuid: "e11561118c2445f29637727c10fbde4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "79667d75809c4babad831658dee965a6"
I20260927 14:55:33.231969 19864 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a67e9674ce154bef9eb3eec2dfe65cef" candidate_uuid: "e11561118c2445f29637727c10fbde4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "681c1b7e05f544be8e7a97864d9bd98d"
I20260927 14:55:33.232028 19730 raft_consensus.cc:3037] T a67e9674ce154bef9eb3eec2dfe65cef P 79667d75809c4babad831658dee965a6 [term 0 FOLLOWER]: Advancing to term 1
I20260927 14:55:33.232120 19864 raft_consensus.cc:3037] T a67e9674ce154bef9eb3eec2dfe65cef P 681c1b7e05f544be8e7a97864d9bd98d [term 0 FOLLOWER]: Advancing to term 1
I20260927 14:55:33.232992 19730 raft_consensus.cc:2454] T a67e9674ce154bef9eb3eec2dfe65cef P 79667d75809c4babad831658dee965a6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e11561118c2445f29637727c10fbde4a in term 1.
I20260927 14:55:33.233192 19864 raft_consensus.cc:2454] T a67e9674ce154bef9eb3eec2dfe65cef P 681c1b7e05f544be8e7a97864d9bd98d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e11561118c2445f29637727c10fbde4a in term 1.
I20260927 14:55:33.233315 19934 leader_election.cc:302] T a67e9674ce154bef9eb3eec2dfe65cef P e11561118c2445f29637727c10fbde4a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 79667d75809c4babad831658dee965a6, e11561118c2445f29637727c10fbde4a; no voters:
I20260927 14:55:33.233479 20067 raft_consensus.cc:2781] T a67e9674ce154bef9eb3eec2dfe65cef P e11561118c2445f29637727c10fbde4a [term 1 FOLLOWER]: Leader election won for term 1
I20260927 14:55:33.233708 20067 raft_consensus.cc:686] T a67e9674ce154bef9eb3eec2dfe65cef P e11561118c2445f29637727c10fbde4a [term 1 LEADER]: Becoming Leader. State: Replica: e11561118c2445f29637727c10fbde4a, State: Running, Role: LEADER
I20260927 14:55:33.234004 20067 consensus_queue.cc:237] T a67e9674ce154bef9eb3eec2dfe65cef P e11561118c2445f29637727c10fbde4a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e11561118c2445f29637727c10fbde4a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46011 } } peers { permanent_uuid: "79667d75809c4babad831658dee965a6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39441 } } peers { permanent_uuid: "681c1b7e05f544be8e7a97864d9bd98d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36349 } }
I20260927 14:55:33.237003 19584 catalog_manager.cc:5495] T a67e9674ce154bef9eb3eec2dfe65cef P e11561118c2445f29637727c10fbde4a reported cstate change: term changed from 0 to 1, leader changed from <none> to e11561118c2445f29637727c10fbde4a (127.0.98.3). New cstate: current_term: 1 leader_uuid: "e11561118c2445f29637727c10fbde4a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e11561118c2445f29637727c10fbde4a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46011 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "79667d75809c4babad831658dee965a6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39441 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "681c1b7e05f544be8e7a97864d9bd98d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36349 } health_report { overall_health: UNKNOWN } } }
I20260927 14:55:33.249869 20057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:55:33.246867 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47842 (local address 127.0.98.3:46011)
0927 14:55:33.246977 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0927 14:55:33.246981 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 14:55:33.246993 (+ 12us) server_negotiation.cc:407] Connection header received
0927 14:55:33.247129 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:55:33.247132 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:55:33.247185 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:55:33.247270 (+ 85us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 14:55:33.247460 (+ 190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:55:33.247924 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:55:33.248697 (+ 773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:55:33.248892 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:55:33.248962 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:55:33.249215 (+ 253us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 14:55:33.249589 (+ 374us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 14:55:33.249674 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0927 14:55:33.249729 (+ 55us) server_negotiation.cc:299] Negotiation successful
0927 14:55:33.249774 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260927 14:55:33.255115 19864 raft_consensus.cc:1261] T a67e9674ce154bef9eb3eec2dfe65cef P 681c1b7e05f544be8e7a97864d9bd98d [term 1 FOLLOWER]: Refusing update from remote peer e11561118c2445f29637727c10fbde4a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 14:55:33.255136 19730 raft_consensus.cc:1261] T a67e9674ce154bef9eb3eec2dfe65cef P 79667d75809c4babad831658dee965a6 [term 1 FOLLOWER]: Refusing update from remote peer e11561118c2445f29637727c10fbde4a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 14:55:33.255759 20067 consensus_queue.cc:1035] T a67e9674ce154bef9eb3eec2dfe65cef P e11561118c2445f29637727c10fbde4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "681c1b7e05f544be8e7a97864d9bd98d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36349 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 14:55:33.255929 20071 consensus_queue.cc:1035] T a67e9674ce154bef9eb3eec2dfe65cef P e11561118c2445f29637727c10fbde4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "79667d75809c4babad831658dee965a6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39441 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 14:55:33.264163 20075 mvcc.cc:205] Tried to move back new op lower bound from 7333973742606745600 to 7333973742529155072. Current Snapshot: MvccSnapshot[applied={T|T < 7333973742606745600}]
I20260927 14:55:33.272375 19584 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53988:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 14:55:33.272547 19584 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53988:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 14:55:33.274932 19584 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b466bd79a44b494592e4140206ed81c6: Sending DeleteTablet for 3 replicas of tablet a67e9674ce154bef9eb3eec2dfe65cef
I20260927 14:55:33.275638 19978 tablet_service.cc:1514] Processing DeleteTablet for tablet a67e9674ce154bef9eb3eec2dfe65cef with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 14:55:33 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:47840
I20260927 14:55:33.275864 19710 tablet_service.cc:1514] Processing DeleteTablet for tablet a67e9674ce154bef9eb3eec2dfe65cef with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 14:55:33 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:56740
I20260927 14:55:33.276260 19844 tablet_service.cc:1514] Processing DeleteTablet for tablet a67e9674ce154bef9eb3eec2dfe65cef with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 14:55:33 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:48110
I20260927 14:55:33.276512 20084 tablet_replica.cc:330] T a67e9674ce154bef9eb3eec2dfe65cef P e11561118c2445f29637727c10fbde4a: stopping tablet replica
I20260927 14:55:33.276566 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6936m5/build/debug/bin/kudu with pid 19647
I20260927 14:55:33.277963 20086 tablet_replica.cc:330] T a67e9674ce154bef9eb3eec2dfe65cef P 681c1b7e05f544be8e7a97864d9bd98d: stopping tablet replica
I20260927 14:55:33.278301 20086 raft_consensus.cc:2229] T a67e9674ce154bef9eb3eec2dfe65cef P 681c1b7e05f544be8e7a97864d9bd98d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 14:55:33.278606 20086 raft_consensus.cc:2258] T a67e9674ce154bef9eb3eec2dfe65cef P 681c1b7e05f544be8e7a97864d9bd98d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 14:55:33.279193 20084 raft_consensus.cc:2229] T a67e9674ce154bef9eb3eec2dfe65cef P e11561118c2445f29637727c10fbde4a [term 1 LEADER]: Raft consensus shutting down.
I20260927 14:55:33.280519 20086 ts_tablet_manager.cc:1905] T a67e9674ce154bef9eb3eec2dfe65cef P 681c1b7e05f544be8e7a97864d9bd98d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 14:55:33.280882 20084 raft_consensus.cc:2258] T a67e9674ce154bef9eb3eec2dfe65cef P e11561118c2445f29637727c10fbde4a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 14:55:33.282347 20084 ts_tablet_manager.cc:1905] T a67e9674ce154bef9eb3eec2dfe65cef P e11561118c2445f29637727c10fbde4a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 14:55:33.283327 20086 ts_tablet_manager.cc:1918] T a67e9674ce154bef9eb3eec2dfe65cef P 681c1b7e05f544be8e7a97864d9bd98d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 14:55:33.283427 20086 log.cc:1196] T a67e9674ce154bef9eb3eec2dfe65cef P 681c1b7e05f544be8e7a97864d9bd98d: Deleting WAL directory at /tmp/dist-test-task6936m5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790520684602251-392-0/minicluster-data/ts-1/wal/wals/a67e9674ce154bef9eb3eec2dfe65cef
I20260927 14:55:33.283773 20086 ts_tablet_manager.cc:1939] T a67e9674ce154bef9eb3eec2dfe65cef P 681c1b7e05f544be8e7a97864d9bd98d: Deleting consensus metadata
W20260927 14:55:33.284785 19572 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:39441 (error 108)
I20260927 14:55:33.285248 19572 catalog_manager.cc:4841] TS 681c1b7e05f544be8e7a97864d9bd98d (127.0.98.2:36349): tablet a67e9674ce154bef9eb3eec2dfe65cef (table test-table [id=4985f57021f644ee8f4352aa26d0a02a]) successfully deleted
I20260927 14:55:33.285542 20084 ts_tablet_manager.cc:1918] T a67e9674ce154bef9eb3eec2dfe65cef P e11561118c2445f29637727c10fbde4a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 14:55:33.285643 20084 log.cc:1196] T a67e9674ce154bef9eb3eec2dfe65cef P e11561118c2445f29637727c10fbde4a: Deleting WAL directory at /tmp/dist-test-task6936m5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790520684602251-392-0/minicluster-data/ts-2/wal/wals/a67e9674ce154bef9eb3eec2dfe65cef
I20260927 14:55:33.285960 20084 ts_tablet_manager.cc:1939] T a67e9674ce154bef9eb3eec2dfe65cef P e11561118c2445f29637727c10fbde4a: Deleting consensus metadata
I20260927 14:55:33.286020 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6936m5/build/debug/bin/kudu with pid 19781
I20260927 14:55:33.286418 20056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:55:33.286081 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:39441 (local address 127.0.0.1:56756)
0927 14:55:33.286262 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0927 14:55:33.286329 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:39441: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98}
W20260927 14:55:33.286801 19572 catalog_manager.cc:4584] TS 79667d75809c4babad831658dee965a6 (127.0.98.1:39441): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet a67e9674ce154bef9eb3eec2dfe65cef: Network error: Client connection negotiation failed: client connection to 127.0.98.1:39441: connect: Connection refused (error 111)
I20260927 14:55:33.287561 19571 catalog_manager.cc:4841] TS e11561118c2445f29637727c10fbde4a (127.0.98.3:46011): tablet a67e9674ce154bef9eb3eec2dfe65cef (table test-table [id=4985f57021f644ee8f4352aa26d0a02a]) successfully deleted
I20260927 14:55:33.292937 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6936m5/build/debug/bin/kudu with pid 19915
I20260927 14:55:33.298770 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6936m5/build/debug/bin/kudu with pid 19553
2026-09-27T14:55:33Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (2249 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-09-27T14:55:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T14:55:33Z Disabled control of system clock
I20260927 14:55:33.328275 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6936m5/build/debug/bin/kudu
/tmp/dist-test-task6936m5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790520684602251-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790520684602251-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790520684602251-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790520684602251-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39383
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:40365
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:39383
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260927 14:55:33.437712 20091 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 14:55:33.438004 20091 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 14:55:33.438040 20091 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 14:55:33.441958 20091 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 14:55:33.442009 20091 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 14:55:33.442024 20091 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 14:55:33.442040 20091 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 14:55:33.442056 20091 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 14:55:33.446867 20091 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40365
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790520684602251-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790520684602251-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:39383
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39383
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790520684602251-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20091
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790520684602251-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:15:17 UTC on ccycloud.quasar-ledxfe.root.comops.site
build id 4676134
I20260927 14:55:33.447885 20091 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 14:55:33.449203 20091 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 14:55:33.457527 20096 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 14:55:33.457527 20099 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 14:55:33.457691 20091 server_base.cc:1034] running on GCE node
W20260927 14:55:33.457576 20097 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 14:55:33.458480 20091 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 14:55:33.459527 20091 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 14:55:33.460760 20091 hybrid_clock.cc:648] HybridClock initialized: now 1790520933460746 us; error 39 us; skew 500 ppm
I20260927 14:55:33.462710 20091 webserver.cc:460] Webserver started at http://127.0.98.62:42795/ using document root <none> and password file <none>
I20260927 14:55:33.463306 20091 fs_manager.cc:359] Metadata directory not provided
I20260927 14:55:33.463397 20091 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 14:55:33.463629 20091 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 14:55:33.465812 20091 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6936m5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790520684602251-392-0/minicluster-data/master-0/data/instance:
uuid: "6532df0ca21842419e5b4a586eea70fb"
format_stamp: "Formatted at 2026-09-27 14:55:33 on dist-test-slave-xk1g"
server_key: "c9dad1bbdbf1dfbffd3724636315c7df"
server_key_iv: "5909aeb091fd396132e595e89c183e80"
server_key_version: "encryptionkey@0"
I20260927 14:55:33.466382 20091 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6936m5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790520684602251-392-0/minicluster-data/master-0/wal/instance:
uuid: "6532df0ca21842419e5b4a586eea70fb"
format_stamp: "Formatted at 2026-09-27 14:55:33 on dist-test-slave-xk1g"
server_key: "c9dad1bbdbf1dfbffd3724636315c7df"
server_key_iv: "5909aeb091fd396132e595e89c183e80"
server_key_version: "encryptionkey@0"
I20260927 14:55:33.470000 20091 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.002s
I20260927 14:55:33.472306 20105 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 14:55:33.473600 20091 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 14:55:33.473744 20091 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790520684602251-392-0/minicluster-data/master-0/data,/tmp/dist-test-task6936m5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790520684602251-392-0/minicluster-data/master-0/wal
uuid: "6532df0ca21842419e5b4a586eea70fb"
format_stamp: "Formatted at 2026-09-27 14:55:33 on dist-test-slave-xk1g"
server_key: "c9dad1bbdbf1dfbffd3724636315c7df"
server_key_iv: "5909aeb091fd396132e595e89c183e80"
server_key_version: "encryptionkey@0"
I20260927 14:55:33.473856 20091 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790520684602251-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790520684602251-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790520684602251-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 14:55:33.484139 20091 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 14:55:33.484794 20091 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 14:55:33.484987 20091 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 14:55:33.491566 20091 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:39383
I20260927 14:55:33.491569 20157 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:39383 every 8 connection(s)
I20260927 14:55:33.492492 20091 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6936m5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790520684602251-392-0/minicluster-data/master-0/data/info.pb
I20260927 14:55:33.493803 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6936m5/build/debug/bin/kudu as pid 20091
I20260927 14:55:33.493947 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6936m5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790520684602251-392-0/minicluster-data/master-0/wal/instance
I20260927 14:55:33.494218 392 external_mini_cluster.cc:1428] Setting key e3f0fb91f1dbf595d71d0e49493fedf5
I20260927 14:55:33.495977 20158 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 14:55:33.502166 20158 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6532df0ca21842419e5b4a586eea70fb: Bootstrap starting.
I20260927 14:55:33.503144 20161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:55:33.495499 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50050 (local address 127.0.98.62:39383)
0927 14:55:33.496129 (+ 630us) server_negotiation.cc:206] Beginning negotiation
0927 14:55:33.496145 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0927 14:55:33.496181 (+ 36us) server_negotiation.cc:407] Connection header received
0927 14:55:33.496812 (+ 631us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:55:33.496827 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:55:33.497285 (+ 458us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:55:33.497561 (+ 276us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:55:33.497876 (+ 315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:55:33.498824 (+ 948us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:55:33.499987 (+ 1163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:55:33.500436 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:55:33.500547 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:55:33.501505 (+ 958us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:55:33.501509 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:55:33.501521 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 14:55:33.501895 (+ 374us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:55:33.501946 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 14:55:33.501965 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:55:33.502040 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0927 14:55:33.502211 (+ 171us) server_negotiation.cc:299] Negotiation successful
0927 14:55:33.502480 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":255,"thread_start_us":136,"threads_started":1}
I20260927 14:55:33.504477 20158 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6532df0ca21842419e5b4a586eea70fb: Neither blocks nor log segments found. Creating new log.
I20260927 14:55:33.505280 20158 log.cc:824] T 00000000000000000000000000000000 P 6532df0ca21842419e5b4a586eea70fb: Log is configured to *not* fsync() on all Append() calls
I20260927 14:55:33.507061 20158 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6532df0ca21842419e5b4a586eea70fb: No bootstrap required, opened a new log
I20260927 14:55:33.510478 20158 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6532df0ca21842419e5b4a586eea70fb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6532df0ca21842419e5b4a586eea70fb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39383 } }
I20260927 14:55:33.510747 20158 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6532df0ca21842419e5b4a586eea70fb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 14:55:33.510854 20158 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6532df0ca21842419e5b4a586eea70fb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6532df0ca21842419e5b4a586eea70fb, State: Initialized, Role: FOLLOWER
I20260927 14:55:33.511323 20158 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6532df0ca21842419e5b4a586eea70fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6532df0ca21842419e5b4a586eea70fb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39383 } }
I20260927 14:55:33.511464 20158 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6532df0ca21842419e5b4a586eea70fb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 14:55:33.511544 20158 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6532df0ca21842419e5b4a586eea70fb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 14:55:33.511642 20158 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6532df0ca21842419e5b4a586eea70fb [term 0 FOLLOWER]: Advancing to term 1
I20260927 14:55:33.512508 20158 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6532df0ca21842419e5b4a586eea70fb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6532df0ca21842419e5b4a586eea70fb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39383 } }
I20260927 14:55:33.512849 20158 leader_election.cc:302] T 00000000000000000000000000000000 P 6532df0ca21842419e5b4a586eea70fb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6532df0ca21842419e5b4a586eea70fb; no voters:
I20260927 14:55:33.513214 20158 leader_election.cc:288] T 00000000000000000000000000000000 P 6532df0ca21842419e5b4a586eea70fb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 14:55:33.513341 20163 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6532df0ca21842419e5b4a586eea70fb [term 1 FOLLOWER]: Leader election won for term 1
I20260927 14:55:33.513568 20163 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6532df0ca21842419e5b4a586eea70fb [term 1 LEADER]: Becoming Leader. State: Replica: 6532df0ca21842419e5b4a586eea70fb, State: Running, Role: LEADER
I20260927 14:55:33.513872 20163 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6532df0ca21842419e5b4a586eea70fb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6532df0ca21842419e5b4a586eea70fb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39383 } }
I20260927 14:55:33.514364 20158 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6532df0ca21842419e5b4a586eea70fb [sys.catalog]: configured and running, proceeding with master startup.
I20260927 14:55:33.515383 20164 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6532df0ca21842419e5b4a586eea70fb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6532df0ca21842419e5b4a586eea70fb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6532df0ca21842419e5b4a586eea70fb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39383 } } }
I20260927 14:55:33.515511 20164 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6532df0ca21842419e5b4a586eea70fb [sys.catalog]: This master's current role is: LEADER
I20260927 14:55:33.515834 20165 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6532df0ca21842419e5b4a586eea70fb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6532df0ca21842419e5b4a586eea70fb. Latest consensus state: current_term: 1 leader_uuid: "6532df0ca21842419e5b4a586eea70fb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6532df0ca21842419e5b4a586eea70fb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39383 } } }
I20260927 14:55:33.515949 20165 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6532df0ca21842419e5b4a586eea70fb [sys.catalog]: This master's current role is: LEADER
I20260927 14:55:33.515897 20172 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 14:55:33.518818 20172 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 14:55:33.524333 20172 catalog_manager.cc:1347] Generated new cluster ID: 63f57c10ed7d422b9f9853db79876254
I20260927 14:55:33.524411 20172 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 14:55:33.545946 20172 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 14:55:33.547169 20172 catalog_manager.cc:1504] Loading token signing keys...
I20260927 14:55:33.550823 20172 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6532df0ca21842419e5b4a586eea70fb: Generated new TSK 0
I20260927 14:55:33.551493 20172 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 14:55:33.559171 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6936m5/build/debug/bin/kudu
/tmp/dist-test-task6936m5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790520684602251-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790520684602251-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790520684602251-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790520684602251-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39383
--builtin_ntp_servers=127.0.98.20:40365
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260927 14:55:33.668493 20182 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 14:55:33.668743 20182 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 14:55:33.668782 20182 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 14:55:33.672806 20182 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 14:55:33.672857 20182 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 14:55:33.672911 20182 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 14:55:33.677801 20182 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40365
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790520684602251-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790520684602251-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790520684602251-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39383
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20182
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790520684602251-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:15:17 UTC on ccycloud.quasar-ledxfe.root.comops.site
build id 4676134
I20260927 14:55:33.678815 20182 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 14:55:33.680128 20182 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 14:55:33.689230 20187 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 14:55:33.689255 20190 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 14:55:33.689258 20188 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 14:55:33.689527 20182 server_base.cc:1034] running on GCE node
I20260927 14:55:33.689904 20182 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 14:55:33.690500 20182 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 14:55:33.691684 20182 hybrid_clock.cc:648] HybridClock initialized: now 1790520933691680 us; error 35 us; skew 500 ppm
I20260927 14:55:33.693434 20182 webserver.cc:460] Webserver started at http://127.0.98.1:37103/ using document root <none> and password file <none>
I20260927 14:55:33.693988 20182 fs_manager.cc:359] Metadata directory not provided
I20260927 14:55:33.694079 20182 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 14:55:33.694311 20182 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 14:55:33.696221 20182 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6936m5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790520684602251-392-0/minicluster-data/ts-0/data/instance:
uuid: "e66ebbf124d84cf1a94ea2c980d748c6"
format_stamp: "Formatted at 2026-09-27 14:55:33 on dist-test-slave-xk1g"
server_key: "0af2d134d4a0a9daa28a00583db8c1da"
server_key_iv: "800d1b742b3747cdd187d6b4fbed85f8"
server_key_version: "encryptionkey@0"
I20260927 14:55:33.696758 20182 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6936m5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790520684602251-392-0/minicluster-data/ts-0/wal/instance:
uuid: "e66ebbf124d84cf1a94ea2c980d748c6"
format_stamp: "Formatted at 2026-09-27 14:55:33 on dist-test-slave-xk1g"
server_key: "0af2d134d4a0a9daa28a00583db8c1da"
server_key_iv: "800d1b742b3747cdd187d6b4fbed85f8"
server_key_version: "encryptionkey@0"
I20260927 14:55:33.700202 20182 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260927 14:55:33.702435 20196 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 14:55:33.703567 20182 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 14:55:33.703694 20182 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790520684602251-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task6936m5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790520684602251-392-0/minicluster-data/ts-0/wal
uuid: "e66ebbf124d84cf1a94ea2c980d748c6"
format_stamp: "Formatted at 2026-09-27 14:55:33 on dist-test-slave-xk1g"
server_key: "0af2d134d4a0a9daa28a00583db8c1da"
server_key_iv: "800d1b742b3747cdd187d6b4fbed85f8"
server_key_version: "encryptionkey@0"
I20260927 14:55:33.703809 20182 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790520684602251-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790520684602251-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790520684602251-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 14:55:33.714221 20182 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 14:55:33.714861 20182 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 14:55:33.715061 20182 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 14:55:33.715626 20182 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 14:55:33.716470 20182 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 14:55:33.716548 20182 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 14:55:33.716617 20182 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 14:55:33.716646 20182 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 14:55:33.725840 20182 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:37365
I20260927 14:55:33.725859 20309 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:37365 every 8 connection(s)
I20260927 14:55:33.726681 20182 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6936m5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790520684602251-392-0/minicluster-data/ts-0/data/info.pb
I20260927 14:55:33.735140 20161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:55:33.728356 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:50135 (local address 127.0.98.62:39383)
0927 14:55:33.728487 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0927 14:55:33.728491 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 14:55:33.729539 (+ 1048us) server_negotiation.cc:407] Connection header received
0927 14:55:33.730832 (+ 1293us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:55:33.730834 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:55:33.730883 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:55:33.730957 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:55:33.731766 (+ 809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:55:33.732300 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:55:33.733382 (+ 1082us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:55:33.733599 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:55:33.733679 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:55:33.734736 (+ 1057us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:55:33.734738 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:55:33.734740 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 14:55:33.734750 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:55:33.734764 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 14:55:33.734771 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:55:33.734832 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0927 14:55:33.734962 (+ 130us) server_negotiation.cc:299] Negotiation successful
0927 14:55:33.735027 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260927 14:55:33.735491 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6936m5/build/debug/bin/kudu as pid 20182
I20260927 14:55:33.735618 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6936m5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790520684602251-392-0/minicluster-data/ts-0/wal/instance
I20260927 14:55:33.735900 392 external_mini_cluster.cc:1428] Setting key 20d8fb1efe8a83f088a02a721792ebf0
I20260927 14:55:33.736071 20312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:55:33.728733 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39383 (local address 127.0.98.1:50135)
0927 14:55:33.729402 (+ 669us) negotiation.cc:107] Waiting for socket to connect
0927 14:55:33.729440 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0927 14:55:33.730656 (+ 1216us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:55:33.731114 (+ 458us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:55:33.731120 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:55:33.731208 (+ 88us) client_negotiation.cc:189] Negotiated authn=SASL
0927 14:55:33.731655 (+ 447us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:55:33.731666 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:55:33.732409 (+ 743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:55:33.732411 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:55:33.733281 (+ 870us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:55:33.733290 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:55:33.733731 (+ 441us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:55:33.733734 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:55:33.733867 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:55:33.734498 (+ 631us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 14:55:33.734499 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0927 14:55:33.734539 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 14:55:33.734600 (+ 61us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 14:55:33.734848 (+ 248us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 14:55:33.734854 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 14:55:33.734858 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 14:55:33.734949 (+ 91us) client_negotiation.cc:240] Negotiation successful
0927 14:55:33.735154 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":241,"thread_start_us":106,"threads_started":1}
I20260927 14:55:33.737409 20310 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39383
I20260927 14:55:33.737640 20310 heartbeater.cc:464] Registering TS with master...
I20260927 14:55:33.737910 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6936m5/build/debug/bin/kudu
/tmp/dist-test-task6936m5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790520684602251-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790520684602251-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790520684602251-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790520684602251-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39383
--builtin_ntp_servers=127.0.98.20:40365
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260927 14:55:33.738236 20310 heartbeater.cc:511] Master 127.0.98.62:39383 requested a full tablet report, sending...
I20260927 14:55:33.739775 20122 ts_manager.cc:194] Registered new tserver with Master: e66ebbf124d84cf1a94ea2c980d748c6 (127.0.98.1:37365)
I20260927 14:55:33.740868 20122 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.1:50135
W20260927 14:55:33.847605 20313 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 14:55:33.847875 20313 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 14:55:33.847915 20313 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 14:55:33.852027 20313 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 14:55:33.852097 20313 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 14:55:33.852154 20313 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 14:55:33.857247 20313 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40365
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790520684602251-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790520684602251-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790520684602251-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39383
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20313
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790520684602251-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:15:17 UTC on ccycloud.quasar-ledxfe.root.comops.site
build id 4676134
I20260927 14:55:33.858319 20313 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 14:55:33.859670 20313 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 14:55:33.869074 20318 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 14:55:33.869102 20319 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 14:55:33.869102 20321 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 14:55:33.869366 20313 server_base.cc:1034] running on GCE node
I20260927 14:55:33.869737 20313 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 14:55:33.870357 20313 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 14:55:33.871512 20313 hybrid_clock.cc:648] HybridClock initialized: now 1790520933871498 us; error 26 us; skew 500 ppm
I20260927 14:55:33.873327 20313 webserver.cc:460] Webserver started at http://127.0.98.2:37795/ using document root <none> and password file <none>
I20260927 14:55:33.873895 20313 fs_manager.cc:359] Metadata directory not provided
I20260927 14:55:33.873955 20313 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 14:55:33.874209 20313 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 14:55:33.876225 20313 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6936m5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790520684602251-392-0/minicluster-data/ts-1/data/instance:
uuid: "cb7842c8ae564181a7185c06fb968d8d"
format_stamp: "Formatted at 2026-09-27 14:55:33 on dist-test-slave-xk1g"
server_key: "7a69d282d0a258206b4a1388a1974a42"
server_key_iv: "1d36b30c76273a1c3804bf53c056ac65"
server_key_version: "encryptionkey@0"
I20260927 14:55:33.876770 20313 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6936m5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790520684602251-392-0/minicluster-data/ts-1/wal/instance:
uuid: "cb7842c8ae564181a7185c06fb968d8d"
format_stamp: "Formatted at 2026-09-27 14:55:33 on dist-test-slave-xk1g"
server_key: "7a69d282d0a258206b4a1388a1974a42"
server_key_iv: "1d36b30c76273a1c3804bf53c056ac65"
server_key_version: "encryptionkey@0"
I20260927 14:55:33.880355 20313 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260927 14:55:33.882686 20327 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 14:55:33.883857 20313 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 14:55:33.883977 20313 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790520684602251-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task6936m5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790520684602251-392-0/minicluster-data/ts-1/wal
uuid: "cb7842c8ae564181a7185c06fb968d8d"
format_stamp: "Formatted at 2026-09-27 14:55:33 on dist-test-slave-xk1g"
server_key: "7a69d282d0a258206b4a1388a1974a42"
server_key_iv: "1d36b30c76273a1c3804bf53c056ac65"
server_key_version: "encryptionkey@0"
I20260927 14:55:33.884079 20313 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790520684602251-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790520684602251-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790520684602251-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 14:55:33.904026 20313 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 14:55:33.904759 20313 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 14:55:33.904969 20313 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 14:55:33.905597 20313 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 14:55:33.906522 20313 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 14:55:33.906594 20313 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 14:55:33.906664 20313 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 14:55:33.906715 20313 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 14:55:33.915644 20313 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:34445
I20260927 14:55:33.915723 20440 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:34445 every 8 connection(s)
I20260927 14:55:33.916610 20313 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6936m5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790520684602251-392-0/minicluster-data/ts-1/data/info.pb
I20260927 14:55:33.925175 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6936m5/build/debug/bin/kudu as pid 20313
I20260927 14:55:33.925310 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6936m5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790520684602251-392-0/minicluster-data/ts-1/wal/instance
I20260927 14:55:33.925580 392 external_mini_cluster.cc:1428] Setting key 5043f8a8fa88720a416039a28bbd6068
I20260927 14:55:33.925860 20161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:55:33.918546 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:48645 (local address 127.0.98.62:39383)
0927 14:55:33.918687 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0927 14:55:33.918690 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 14:55:33.919769 (+ 1079us) server_negotiation.cc:407] Connection header received
0927 14:55:33.921200 (+ 1431us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:55:33.921203 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:55:33.921261 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:55:33.921376 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:55:33.922251 (+ 875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:55:33.922777 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:55:33.923907 (+ 1130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:55:33.924129 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:55:33.924241 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:55:33.925367 (+ 1126us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:55:33.925375 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:55:33.925377 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 14:55:33.925388 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:55:33.925406 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 14:55:33.925414 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:55:33.925492 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0927 14:55:33.925659 (+ 167us) server_negotiation.cc:299] Negotiation successful
0927 14:55:33.925719 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260927 14:55:33.926851 20443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:55:33.918974 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39383 (local address 127.0.98.2:48645)
0927 14:55:33.919627 (+ 653us) negotiation.cc:107] Waiting for socket to connect
0927 14:55:33.919666 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0927 14:55:33.920938 (+ 1272us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:55:33.921545 (+ 607us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:55:33.921551 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:55:33.921634 (+ 83us) client_negotiation.cc:189] Negotiated authn=SASL
0927 14:55:33.922106 (+ 472us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:55:33.922117 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:55:33.922901 (+ 784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:55:33.922905 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:55:33.923769 (+ 864us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:55:33.923787 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:55:33.924278 (+ 491us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:55:33.924282 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:55:33.924463 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:55:33.925103 (+ 640us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 14:55:33.925105 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0927 14:55:33.925143 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 14:55:33.925201 (+ 58us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 14:55:33.925507 (+ 306us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 14:55:33.925517 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 14:55:33.925524 (+ 7us) client_negotiation.cc:769] Sending connection context
0927 14:55:33.925656 (+ 132us) client_negotiation.cc:240] Negotiation successful
0927 14:55:33.925939 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":270,"thread_start_us":119,"threads_started":1}
I20260927 14:55:33.927580 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6936m5/build/debug/bin/kudu
/tmp/dist-test-task6936m5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790520684602251-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790520684602251-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790520684602251-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790520684602251-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39383
--builtin_ntp_servers=127.0.98.20:40365
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260927 14:55:33.928391 20441 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39383
I20260927 14:55:33.928633 20441 heartbeater.cc:464] Registering TS with master...
I20260927 14:55:33.929282 20441 heartbeater.cc:511] Master 127.0.98.62:39383 requested a full tablet report, sending...
I20260927 14:55:33.930404 20122 ts_manager.cc:194] Registered new tserver with Master: cb7842c8ae564181a7185c06fb968d8d (127.0.98.2:34445)
I20260927 14:55:33.931243 20122 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.2:48645
W20260927 14:55:34.037135 20444 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 14:55:34.037390 20444 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 14:55:34.037427 20444 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 14:55:34.041373 20444 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 14:55:34.041427 20444 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 14:55:34.041481 20444 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 14:55:34.046118 20444 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40365
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790520684602251-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790520684602251-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790520684602251-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39383
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20444
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790520684602251-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:15:17 UTC on ccycloud.quasar-ledxfe.root.comops.site
build id 4676134
I20260927 14:55:34.047111 20444 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 14:55:34.048374 20444 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 14:55:34.057605 20449 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 14:55:34.057597 20452 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 14:55:34.057593 20450 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 14:55:34.057837 20444 server_base.cc:1034] running on GCE node
I20260927 14:55:34.058321 20444 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 14:55:34.058920 20444 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 14:55:34.060076 20444 hybrid_clock.cc:648] HybridClock initialized: now 1790520934060060 us; error 27 us; skew 500 ppm
I20260927 14:55:34.061920 20444 webserver.cc:460] Webserver started at http://127.0.98.3:43537/ using document root <none> and password file <none>
I20260927 14:55:34.062479 20444 fs_manager.cc:359] Metadata directory not provided
I20260927 14:55:34.062541 20444 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 14:55:34.062783 20444 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 14:55:34.064832 20444 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6936m5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790520684602251-392-0/minicluster-data/ts-2/data/instance:
uuid: "1e5a252a48594ebeb458fad76c59dcb6"
format_stamp: "Formatted at 2026-09-27 14:55:34 on dist-test-slave-xk1g"
server_key: "c05ac035598907fa418a0f6cd13ef7a1"
server_key_iv: "0e359171c1bc6aa27ca494f6e04eca29"
server_key_version: "encryptionkey@0"
I20260927 14:55:34.065436 20444 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6936m5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790520684602251-392-0/minicluster-data/ts-2/wal/instance:
uuid: "1e5a252a48594ebeb458fad76c59dcb6"
format_stamp: "Formatted at 2026-09-27 14:55:34 on dist-test-slave-xk1g"
server_key: "c05ac035598907fa418a0f6cd13ef7a1"
server_key_iv: "0e359171c1bc6aa27ca494f6e04eca29"
server_key_version: "encryptionkey@0"
I20260927 14:55:34.068914 20444 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260927 14:55:34.071255 20458 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 14:55:34.072431 20444 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 14:55:34.072572 20444 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790520684602251-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task6936m5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790520684602251-392-0/minicluster-data/ts-2/wal
uuid: "1e5a252a48594ebeb458fad76c59dcb6"
format_stamp: "Formatted at 2026-09-27 14:55:34 on dist-test-slave-xk1g"
server_key: "c05ac035598907fa418a0f6cd13ef7a1"
server_key_iv: "0e359171c1bc6aa27ca494f6e04eca29"
server_key_version: "encryptionkey@0"
I20260927 14:55:34.072685 20444 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790520684602251-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790520684602251-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790520684602251-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 14:55:34.084009 20444 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 14:55:34.084652 20444 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 14:55:34.084837 20444 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 14:55:34.085449 20444 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 14:55:34.086334 20444 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 14:55:34.086405 20444 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 14:55:34.086475 20444 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 14:55:34.086527 20444 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 14:55:34.095122 20444 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:40021
I20260927 14:55:34.095144 20571 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:40021 every 8 connection(s)
I20260927 14:55:34.096000 20444 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6936m5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790520684602251-392-0/minicluster-data/ts-2/data/info.pb
I20260927 14:55:34.104466 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6936m5/build/debug/bin/kudu as pid 20444
I20260927 14:55:34.104586 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6936m5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790520684602251-392-0/minicluster-data/ts-2/wal/instance
I20260927 14:55:34.104655 20161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:55:34.097769 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:45763 (local address 127.0.98.62:39383)
0927 14:55:34.097910 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0927 14:55:34.097914 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 14:55:34.098922 (+ 1008us) server_negotiation.cc:407] Connection header received
0927 14:55:34.100214 (+ 1292us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:55:34.100216 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:55:34.100262 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:55:34.100342 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:55:34.101274 (+ 932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:55:34.101788 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:55:34.102859 (+ 1071us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:55:34.103063 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:55:34.103139 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:55:34.104127 (+ 988us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:55:34.104130 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:55:34.104133 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 14:55:34.104144 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:55:34.104186 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 14:55:34.104201 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:55:34.104289 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0927 14:55:34.104433 (+ 144us) server_negotiation.cc:299] Negotiation successful
0927 14:55:34.104498 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260927 14:55:34.104883 392 external_mini_cluster.cc:1428] Setting key ea70ea1f73a32dd06ba02546fb14dd8b
I20260927 14:55:34.105641 20574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:55:34.098164 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39383 (local address 127.0.98.3:45763)
0927 14:55:34.098787 (+ 623us) negotiation.cc:107] Waiting for socket to connect
0927 14:55:34.098824 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0927 14:55:34.100033 (+ 1209us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:55:34.100499 (+ 466us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:55:34.100506 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:55:34.100589 (+ 83us) client_negotiation.cc:189] Negotiated authn=SASL
0927 14:55:34.101144 (+ 555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:55:34.101159 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:55:34.101894 (+ 735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:55:34.101896 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:55:34.102753 (+ 857us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:55:34.102761 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:55:34.103183 (+ 422us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:55:34.103186 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:55:34.103316 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:55:34.103885 (+ 569us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 14:55:34.103886 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0927 14:55:34.103925 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 14:55:34.103986 (+ 61us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 14:55:34.104284 (+ 298us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 14:55:34.104290 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 14:55:34.104293 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 14:55:34.104395 (+ 102us) client_negotiation.cc:240] Negotiation successful
0927 14:55:34.104662 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":248,"thread_start_us":113,"threads_started":1}
I20260927 14:55:34.106707 20572 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39383
I20260927 14:55:34.106973 20572 heartbeater.cc:464] Registering TS with master...
I20260927 14:55:34.107474 20572 heartbeater.cc:511] Master 127.0.98.62:39383 requested a full tablet report, sending...
I20260927 14:55:34.108497 20121 ts_manager.cc:194] Registered new tserver with Master: 1e5a252a48594ebeb458fad76c59dcb6 (127.0.98.3:40021)
I20260927 14:55:34.109140 20121 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.3:45763
I20260927 14:55:34.118803 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 14:55:34.124135 20161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:55:34.120770 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50064 (local address 127.0.98.62:39383)
0927 14:55:34.120916 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0927 14:55:34.120919 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 14:55:34.121141 (+ 222us) server_negotiation.cc:407] Connection header received
0927 14:55:34.121244 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:55:34.121246 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:55:34.121293 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:55:34.121358 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:55:34.121562 (+ 204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:55:34.122057 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:55:34.122832 (+ 775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:55:34.123033 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:55:34.123106 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:55:34.123799 (+ 693us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:55:34.123802 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:55:34.123804 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 14:55:34.123815 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:55:34.123833 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 14:55:34.123840 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:55:34.123897 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0927 14:55:34.123987 (+ 90us) server_negotiation.cc:299] Negotiation successful
0927 14:55:34.124031 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260927 14:55:34.126864 20121 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50064:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 14:55:34.129907 20121 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 14:55:34.143308 20584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:55:34.137951 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43280 (local address 127.0.98.2:34445)
0927 14:55:34.138272 (+ 321us) server_negotiation.cc:206] Beginning negotiation
0927 14:55:34.138283 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 14:55:34.138314 (+ 31us) server_negotiation.cc:407] Connection header received
0927 14:55:34.138462 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:55:34.138470 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:55:34.138709 (+ 239us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:55:34.138874 (+ 165us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 14:55:34.139369 (+ 495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:55:34.140511 (+ 1142us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:55:34.141745 (+ 1234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:55:34.142487 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:55:34.142576 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:55:34.142716 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0927 14:55:34.143047 (+ 331us) server_negotiation.cc:299] Negotiation successful
0927 14:55:34.143145 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":225,"thread_start_us":147,"threads_started":1}
I20260927 14:55:34.143294 20583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:55:34.137803 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:34445 (local address 127.0.0.1:43280)
0927 14:55:34.138123 (+ 320us) negotiation.cc:107] Waiting for socket to connect
0927 14:55:34.138168 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0927 14:55:34.138295 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:55:34.139053 (+ 758us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:55:34.139058 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:55:34.139076 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 14:55:34.139226 (+ 150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:55:34.139233 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:55:34.140619 (+ 1386us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:55:34.140624 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:55:34.141597 (+ 973us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:55:34.141608 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:55:34.142600 (+ 992us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:55:34.142604 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:55:34.142862 (+ 258us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:55:34.142897 (+ 35us) client_negotiation.cc:769] Sending connection context
0927 14:55:34.143014 (+ 117us) client_negotiation.cc:240] Negotiation successful
0927 14:55:34.143133 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":158,"mutex_wait_us":100,"thread_start_us":96,"threads_started":1}
I20260927 14:55:34.144572 20587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:55:34.138837 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37365 (local address 127.0.0.1:42820)
0927 14:55:34.139450 (+ 613us) negotiation.cc:107] Waiting for socket to connect
0927 14:55:34.139462 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 14:55:34.139540 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:55:34.140145 (+ 605us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:55:34.140147 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:55:34.140159 (+ 12us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 14:55:34.140319 (+ 160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:55:34.140329 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:55:34.141753 (+ 1424us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:55:34.141755 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:55:34.143133 (+ 1378us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:55:34.143143 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:55:34.144187 (+ 1044us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:55:34.144191 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:55:34.144272 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:55:34.144313 (+ 41us) client_negotiation.cc:769] Sending connection context
0927 14:55:34.144380 (+ 67us) client_negotiation.cc:240] Negotiation successful
0927 14:55:34.144444 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":552,"mutex_wait_us":136,"spinlock_wait_cycles":1408,"thread_start_us":74,"threads_started":1}
I20260927 14:55:34.144687 20588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:55:34.139229 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42820 (local address 127.0.98.1:37365)
0927 14:55:34.139766 (+ 537us) server_negotiation.cc:206] Beginning negotiation
0927 14:55:34.139774 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 14:55:34.139791 (+ 17us) server_negotiation.cc:407] Connection header received
0927 14:55:34.139850 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:55:34.139853 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:55:34.139990 (+ 137us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:55:34.140293 (+ 303us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 14:55:34.140430 (+ 137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:55:34.141613 (+ 1183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:55:34.143277 (+ 1664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:55:34.144047 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:55:34.144180 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:55:34.144379 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0927 14:55:34.144433 (+ 54us) server_negotiation.cc:299] Negotiation successful
0927 14:55:34.144517 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":456,"thread_start_us":157,"threads_started":1}
I20260927 14:55:34.145816 20585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:55:34.138548 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:40021 (local address 127.0.0.1:54482)
0927 14:55:34.138788 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0927 14:55:34.138804 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 14:55:34.138879 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:55:34.140121 (+ 1242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:55:34.140126 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:55:34.140146 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 14:55:34.140292 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:55:34.140299 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:55:34.141745 (+ 1446us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:55:34.141748 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:55:34.144027 (+ 2279us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:55:34.144038 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:55:34.145438 (+ 1400us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:55:34.145444 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:55:34.145530 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:55:34.145553 (+ 23us) client_negotiation.cc:769] Sending connection context
0927 14:55:34.145618 (+ 65us) client_negotiation.cc:240] Negotiation successful
0927 14:55:34.145679 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":166,"mutex_wait_us":88,"thread_start_us":80,"threads_started":1}
I20260927 14:55:34.146402 20375 tablet_service.cc:1467] Processing CreateTablet for tablet 9e852e63dd3d4c48955a98426d3b12ab (DEFAULT_TABLE table=test-table [id=aad43c09acfa466bb527603d4ce18351]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 14:55:34.146950 20244 tablet_service.cc:1467] Processing CreateTablet for tablet 9e852e63dd3d4c48955a98426d3b12ab (DEFAULT_TABLE table=test-table [id=aad43c09acfa466bb527603d4ce18351]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 14:55:34.147336 20375 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9e852e63dd3d4c48955a98426d3b12ab. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 14:55:34.147802 20244 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9e852e63dd3d4c48955a98426d3b12ab. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 14:55:34.152848 20589 tablet_bootstrap.cc:492] T 9e852e63dd3d4c48955a98426d3b12ab P e66ebbf124d84cf1a94ea2c980d748c6: Bootstrap starting.
I20260927 14:55:34.152848 20590 tablet_bootstrap.cc:492] T 9e852e63dd3d4c48955a98426d3b12ab P cb7842c8ae564181a7185c06fb968d8d: Bootstrap starting.
I20260927 14:55:34.154569 20586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:55:34.138719 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54482 (local address 127.0.98.3:40021)
0927 14:55:34.139703 (+ 984us) server_negotiation.cc:206] Beginning negotiation
0927 14:55:34.139713 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 14:55:34.139737 (+ 24us) server_negotiation.cc:407] Connection header received
0927 14:55:34.139804 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:55:34.139809 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:55:34.139970 (+ 161us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:55:34.140117 (+ 147us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 14:55:34.140425 (+ 308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:55:34.141628 (+ 1203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:55:34.144170 (+ 2542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:55:34.145311 (+ 1141us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:55:34.154068 (+ 8757us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:55:34.154263 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0927 14:55:34.154318 (+ 55us) server_negotiation.cc:299] Negotiation successful
0927 14:55:34.154414 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":880,"thread_start_us":131,"threads_started":1}
I20260927 14:55:34.154914 20590 tablet_bootstrap.cc:654] T 9e852e63dd3d4c48955a98426d3b12ab P cb7842c8ae564181a7185c06fb968d8d: Neither blocks nor log segments found. Creating new log.
I20260927 14:55:34.154938 20589 tablet_bootstrap.cc:654] T 9e852e63dd3d4c48955a98426d3b12ab P e66ebbf124d84cf1a94ea2c980d748c6: Neither blocks nor log segments found. Creating new log.
I20260927 14:55:34.155668 20589 log.cc:824] T 9e852e63dd3d4c48955a98426d3b12ab P e66ebbf124d84cf1a94ea2c980d748c6: Log is configured to *not* fsync() on all Append() calls
I20260927 14:55:34.155711 20590 log.cc:824] T 9e852e63dd3d4c48955a98426d3b12ab P cb7842c8ae564181a7185c06fb968d8d: Log is configured to *not* fsync() on all Append() calls
I20260927 14:55:34.157537 20590 tablet_bootstrap.cc:492] T 9e852e63dd3d4c48955a98426d3b12ab P cb7842c8ae564181a7185c06fb968d8d: No bootstrap required, opened a new log
I20260927 14:55:34.157639 20589 tablet_bootstrap.cc:492] T 9e852e63dd3d4c48955a98426d3b12ab P e66ebbf124d84cf1a94ea2c980d748c6: No bootstrap required, opened a new log
I20260927 14:55:34.157213 20506 tablet_service.cc:1467] Processing CreateTablet for tablet 9e852e63dd3d4c48955a98426d3b12ab (DEFAULT_TABLE table=test-table [id=aad43c09acfa466bb527603d4ce18351]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 14:55:34.157733 20590 ts_tablet_manager.cc:1397] T 9e852e63dd3d4c48955a98426d3b12ab P cb7842c8ae564181a7185c06fb968d8d: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260927 14:55:34.157809 20589 ts_tablet_manager.cc:1397] T 9e852e63dd3d4c48955a98426d3b12ab P e66ebbf124d84cf1a94ea2c980d748c6: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260927 14:55:34.158334 20506 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9e852e63dd3d4c48955a98426d3b12ab. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 14:55:34.160959 20590 raft_consensus.cc:348] T 9e852e63dd3d4c48955a98426d3b12ab P cb7842c8ae564181a7185c06fb968d8d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb7842c8ae564181a7185c06fb968d8d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34445 } } peers { permanent_uuid: "1e5a252a48594ebeb458fad76c59dcb6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40021 } } peers { permanent_uuid: "e66ebbf124d84cf1a94ea2c980d748c6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37365 } }
I20260927 14:55:34.161262 20590 raft_consensus.cc:374] T 9e852e63dd3d4c48955a98426d3b12ab P cb7842c8ae564181a7185c06fb968d8d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 14:55:34.161316 20590 raft_consensus.cc:729] T 9e852e63dd3d4c48955a98426d3b12ab P cb7842c8ae564181a7185c06fb968d8d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cb7842c8ae564181a7185c06fb968d8d, State: Initialized, Role: FOLLOWER
I20260927 14:55:34.161765 20590 consensus_queue.cc:260] T 9e852e63dd3d4c48955a98426d3b12ab P cb7842c8ae564181a7185c06fb968d8d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb7842c8ae564181a7185c06fb968d8d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34445 } } peers { permanent_uuid: "1e5a252a48594ebeb458fad76c59dcb6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40021 } } peers { permanent_uuid: "e66ebbf124d84cf1a94ea2c980d748c6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37365 } }
I20260927 14:55:34.161947 20589 raft_consensus.cc:348] T 9e852e63dd3d4c48955a98426d3b12ab P e66ebbf124d84cf1a94ea2c980d748c6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb7842c8ae564181a7185c06fb968d8d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34445 } } peers { permanent_uuid: "1e5a252a48594ebeb458fad76c59dcb6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40021 } } peers { permanent_uuid: "e66ebbf124d84cf1a94ea2c980d748c6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37365 } }
I20260927 14:55:34.162199 20589 raft_consensus.cc:374] T 9e852e63dd3d4c48955a98426d3b12ab P e66ebbf124d84cf1a94ea2c980d748c6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 14:55:34.162245 20589 raft_consensus.cc:729] T 9e852e63dd3d4c48955a98426d3b12ab P e66ebbf124d84cf1a94ea2c980d748c6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e66ebbf124d84cf1a94ea2c980d748c6, State: Initialized, Role: FOLLOWER
I20260927 14:55:34.162684 20589 consensus_queue.cc:260] T 9e852e63dd3d4c48955a98426d3b12ab P e66ebbf124d84cf1a94ea2c980d748c6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb7842c8ae564181a7185c06fb968d8d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34445 } } peers { permanent_uuid: "1e5a252a48594ebeb458fad76c59dcb6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40021 } } peers { permanent_uuid: "e66ebbf124d84cf1a94ea2c980d748c6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37365 } }
I20260927 14:55:34.162787 20590 ts_tablet_manager.cc:1428] T 9e852e63dd3d4c48955a98426d3b12ab P cb7842c8ae564181a7185c06fb968d8d: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260927 14:55:34.163125 20441 heartbeater.cc:503] Master 127.0.98.62:39383 was elected leader, sending a full tablet report...
I20260927 14:55:34.163601 20589 ts_tablet_manager.cc:1428] T 9e852e63dd3d4c48955a98426d3b12ab P e66ebbf124d84cf1a94ea2c980d748c6: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260927 14:55:34.163940 20310 heartbeater.cc:503] Master 127.0.98.62:39383 was elected leader, sending a full tablet report...
I20260927 14:55:34.165503 20595 tablet_bootstrap.cc:492] T 9e852e63dd3d4c48955a98426d3b12ab P 1e5a252a48594ebeb458fad76c59dcb6: Bootstrap starting.
W20260927 14:55:34.167539 20442 tablet.cc:2369] T 9e852e63dd3d4c48955a98426d3b12ab P cb7842c8ae564181a7185c06fb968d8d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 14:55:34.168126 20595 tablet_bootstrap.cc:654] T 9e852e63dd3d4c48955a98426d3b12ab P 1e5a252a48594ebeb458fad76c59dcb6: Neither blocks nor log segments found. Creating new log.
I20260927 14:55:34.168992 20595 log.cc:824] T 9e852e63dd3d4c48955a98426d3b12ab P 1e5a252a48594ebeb458fad76c59dcb6: Log is configured to *not* fsync() on all Append() calls
I20260927 14:55:34.170639 20595 tablet_bootstrap.cc:492] T 9e852e63dd3d4c48955a98426d3b12ab P 1e5a252a48594ebeb458fad76c59dcb6: No bootstrap required, opened a new log
I20260927 14:55:34.170848 20595 ts_tablet_manager.cc:1397] T 9e852e63dd3d4c48955a98426d3b12ab P 1e5a252a48594ebeb458fad76c59dcb6: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260927 14:55:34.173904 20595 raft_consensus.cc:348] T 9e852e63dd3d4c48955a98426d3b12ab P 1e5a252a48594ebeb458fad76c59dcb6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb7842c8ae564181a7185c06fb968d8d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34445 } } peers { permanent_uuid: "1e5a252a48594ebeb458fad76c59dcb6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40021 } } peers { permanent_uuid: "e66ebbf124d84cf1a94ea2c980d748c6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37365 } }
I20260927 14:55:34.174099 20595 raft_consensus.cc:374] T 9e852e63dd3d4c48955a98426d3b12ab P 1e5a252a48594ebeb458fad76c59dcb6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 14:55:34.174178 20595 raft_consensus.cc:729] T 9e852e63dd3d4c48955a98426d3b12ab P 1e5a252a48594ebeb458fad76c59dcb6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e5a252a48594ebeb458fad76c59dcb6, State: Initialized, Role: FOLLOWER
I20260927 14:55:34.174574 20595 consensus_queue.cc:260] T 9e852e63dd3d4c48955a98426d3b12ab P 1e5a252a48594ebeb458fad76c59dcb6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb7842c8ae564181a7185c06fb968d8d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34445 } } peers { permanent_uuid: "1e5a252a48594ebeb458fad76c59dcb6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40021 } } peers { permanent_uuid: "e66ebbf124d84cf1a94ea2c980d748c6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37365 } }
I20260927 14:55:34.175297 20572 heartbeater.cc:503] Master 127.0.98.62:39383 was elected leader, sending a full tablet report...
I20260927 14:55:34.175589 20595 ts_tablet_manager.cc:1428] T 9e852e63dd3d4c48955a98426d3b12ab P 1e5a252a48594ebeb458fad76c59dcb6: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
W20260927 14:55:34.227813 20311 tablet.cc:2369] T 9e852e63dd3d4c48955a98426d3b12ab P e66ebbf124d84cf1a94ea2c980d748c6: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 14:55:34.346915 20573 tablet.cc:2369] T 9e852e63dd3d4c48955a98426d3b12ab P 1e5a252a48594ebeb458fad76c59dcb6: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 14:55:34.351341 20597 raft_consensus.cc:482] T 9e852e63dd3d4c48955a98426d3b12ab P 1e5a252a48594ebeb458fad76c59dcb6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 14:55:34.351555 20597 raft_consensus.cc:504] T 9e852e63dd3d4c48955a98426d3b12ab P 1e5a252a48594ebeb458fad76c59dcb6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb7842c8ae564181a7185c06fb968d8d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34445 } } peers { permanent_uuid: "1e5a252a48594ebeb458fad76c59dcb6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40021 } } peers { permanent_uuid: "e66ebbf124d84cf1a94ea2c980d748c6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37365 } }
I20260927 14:55:34.352654 20597 leader_election.cc:288] T 9e852e63dd3d4c48955a98426d3b12ab P 1e5a252a48594ebeb458fad76c59dcb6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cb7842c8ae564181a7185c06fb968d8d (127.0.98.2:34445), e66ebbf124d84cf1a94ea2c980d748c6 (127.0.98.1:37365)
I20260927 14:55:34.356658 20574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:55:34.352951 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:34445 (local address 127.0.98.3:52169)
0927 14:55:34.353094 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0927 14:55:34.353112 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 14:55:34.353225 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:55:34.353566 (+ 341us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:55:34.353570 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:55:34.353590 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 14:55:34.353734 (+ 144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:55:34.353743 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:55:34.354636 (+ 893us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:55:34.354640 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:55:34.355510 (+ 870us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:55:34.355519 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:55:34.356232 (+ 713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:55:34.356235 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:55:34.356338 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:55:34.356375 (+ 37us) client_negotiation.cc:769] Sending connection context
0927 14:55:34.356448 (+ 73us) client_negotiation.cc:240] Negotiation successful
0927 14:55:34.356522 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":50,"mutex_wait_us":1}
I20260927 14:55:34.356683 20584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:55:34.353108 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:52169 (local address 127.0.98.2:34445)
0927 14:55:34.353222 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0927 14:55:34.353229 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 14:55:34.353249 (+ 20us) server_negotiation.cc:407] Connection header received
0927 14:55:34.353357 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:55:34.353360 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:55:34.353413 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:55:34.353504 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 14:55:34.353864 (+ 360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:55:34.354522 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:55:34.355628 (+ 1106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:55:34.356107 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:55:34.356187 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:55:34.356235 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0927 14:55:34.356473 (+ 238us) server_negotiation.cc:299] Negotiation successful
0927 14:55:34.356544 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260927 14:55:34.356906 20588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:55:34.353409 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:40755 (local address 127.0.98.1:37365)
0927 14:55:34.353545 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0927 14:55:34.353549 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 14:55:34.353737 (+ 188us) server_negotiation.cc:407] Connection header received
0927 14:55:34.353798 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:55:34.353804 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:55:34.353864 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:55:34.353926 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 14:55:34.354221 (+ 295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:55:34.354758 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:55:34.355836 (+ 1078us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:55:34.356315 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:55:34.356424 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:55:34.356482 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0927 14:55:34.356700 (+ 218us) server_negotiation.cc:299] Negotiation successful
0927 14:55:34.356773 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260927 14:55:34.356907 20598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:55:34.353275 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37365 (local address 127.0.98.3:40755)
0927 14:55:34.353630 (+ 355us) negotiation.cc:107] Waiting for socket to connect
0927 14:55:34.353646 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 14:55:34.353733 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:55:34.353963 (+ 230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:55:34.353966 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:55:34.353982 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 14:55:34.354102 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:55:34.354109 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:55:34.354869 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:55:34.354871 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:55:34.355712 (+ 841us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:55:34.355721 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:55:34.356468 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:55:34.356471 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:55:34.356564 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:55:34.356586 (+ 22us) client_negotiation.cc:769] Sending connection context
0927 14:55:34.356663 (+ 77us) client_negotiation.cc:240] Negotiation successful
0927 14:55:34.356735 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":289,"mutex_wait_us":24,"thread_start_us":123,"threads_started":1}
I20260927 14:55:34.357394 20395 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e852e63dd3d4c48955a98426d3b12ab" candidate_uuid: "1e5a252a48594ebeb458fad76c59dcb6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cb7842c8ae564181a7185c06fb968d8d" is_pre_election: true
I20260927 14:55:34.357609 20264 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e852e63dd3d4c48955a98426d3b12ab" candidate_uuid: "1e5a252a48594ebeb458fad76c59dcb6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e66ebbf124d84cf1a94ea2c980d748c6" is_pre_election: true
I20260927 14:55:34.357699 20395 raft_consensus.cc:2454] T 9e852e63dd3d4c48955a98426d3b12ab P cb7842c8ae564181a7185c06fb968d8d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e5a252a48594ebeb458fad76c59dcb6 in term 0.
I20260927 14:55:34.357867 20264 raft_consensus.cc:2454] T 9e852e63dd3d4c48955a98426d3b12ab P e66ebbf124d84cf1a94ea2c980d748c6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e5a252a48594ebeb458fad76c59dcb6 in term 0.
I20260927 14:55:34.358188 20459 leader_election.cc:302] T 9e852e63dd3d4c48955a98426d3b12ab P 1e5a252a48594ebeb458fad76c59dcb6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e5a252a48594ebeb458fad76c59dcb6, cb7842c8ae564181a7185c06fb968d8d; no voters:
I20260927 14:55:34.358404 20597 raft_consensus.cc:2781] T 9e852e63dd3d4c48955a98426d3b12ab P 1e5a252a48594ebeb458fad76c59dcb6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 14:55:34.358501 20597 raft_consensus.cc:482] T 9e852e63dd3d4c48955a98426d3b12ab P 1e5a252a48594ebeb458fad76c59dcb6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 14:55:34.358558 20597 raft_consensus.cc:3037] T 9e852e63dd3d4c48955a98426d3b12ab P 1e5a252a48594ebeb458fad76c59dcb6 [term 0 FOLLOWER]: Advancing to term 1
I20260927 14:55:34.359522 20597 raft_consensus.cc:504] T 9e852e63dd3d4c48955a98426d3b12ab P 1e5a252a48594ebeb458fad76c59dcb6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb7842c8ae564181a7185c06fb968d8d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34445 } } peers { permanent_uuid: "1e5a252a48594ebeb458fad76c59dcb6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40021 } } peers { permanent_uuid: "e66ebbf124d84cf1a94ea2c980d748c6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37365 } }
I20260927 14:55:34.359872 20597 leader_election.cc:288] T 9e852e63dd3d4c48955a98426d3b12ab P 1e5a252a48594ebeb458fad76c59dcb6 [CANDIDATE]: Term 1 election: Requested vote from peers cb7842c8ae564181a7185c06fb968d8d (127.0.98.2:34445), e66ebbf124d84cf1a94ea2c980d748c6 (127.0.98.1:37365)
I20260927 14:55:34.360179 20395 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e852e63dd3d4c48955a98426d3b12ab" candidate_uuid: "1e5a252a48594ebeb458fad76c59dcb6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cb7842c8ae564181a7185c06fb968d8d"
I20260927 14:55:34.360239 20264 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e852e63dd3d4c48955a98426d3b12ab" candidate_uuid: "1e5a252a48594ebeb458fad76c59dcb6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e66ebbf124d84cf1a94ea2c980d748c6"
I20260927 14:55:34.360306 20395 raft_consensus.cc:3037] T 9e852e63dd3d4c48955a98426d3b12ab P cb7842c8ae564181a7185c06fb968d8d [term 0 FOLLOWER]: Advancing to term 1
I20260927 14:55:34.360338 20264 raft_consensus.cc:3037] T 9e852e63dd3d4c48955a98426d3b12ab P e66ebbf124d84cf1a94ea2c980d748c6 [term 0 FOLLOWER]: Advancing to term 1
I20260927 14:55:34.361577 20264 raft_consensus.cc:2454] T 9e852e63dd3d4c48955a98426d3b12ab P e66ebbf124d84cf1a94ea2c980d748c6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e5a252a48594ebeb458fad76c59dcb6 in term 1.
I20260927 14:55:34.361577 20395 raft_consensus.cc:2454] T 9e852e63dd3d4c48955a98426d3b12ab P cb7842c8ae564181a7185c06fb968d8d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e5a252a48594ebeb458fad76c59dcb6 in term 1.
I20260927 14:55:34.361869 20459 leader_election.cc:302] T 9e852e63dd3d4c48955a98426d3b12ab P 1e5a252a48594ebeb458fad76c59dcb6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e5a252a48594ebeb458fad76c59dcb6, cb7842c8ae564181a7185c06fb968d8d; no voters:
I20260927 14:55:34.362056 20597 raft_consensus.cc:2781] T 9e852e63dd3d4c48955a98426d3b12ab P 1e5a252a48594ebeb458fad76c59dcb6 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 14:55:34.362283 20597 raft_consensus.cc:686] T 9e852e63dd3d4c48955a98426d3b12ab P 1e5a252a48594ebeb458fad76c59dcb6 [term 1 LEADER]: Becoming Leader. State: Replica: 1e5a252a48594ebeb458fad76c59dcb6, State: Running, Role: LEADER
I20260927 14:55:34.362596 20597 consensus_queue.cc:237] T 9e852e63dd3d4c48955a98426d3b12ab P 1e5a252a48594ebeb458fad76c59dcb6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb7842c8ae564181a7185c06fb968d8d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34445 } } peers { permanent_uuid: "1e5a252a48594ebeb458fad76c59dcb6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40021 } } peers { permanent_uuid: "e66ebbf124d84cf1a94ea2c980d748c6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37365 } }
I20260927 14:55:34.365733 20121 catalog_manager.cc:5495] T 9e852e63dd3d4c48955a98426d3b12ab P 1e5a252a48594ebeb458fad76c59dcb6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1e5a252a48594ebeb458fad76c59dcb6 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "1e5a252a48594ebeb458fad76c59dcb6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb7842c8ae564181a7185c06fb968d8d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34445 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1e5a252a48594ebeb458fad76c59dcb6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40021 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e66ebbf124d84cf1a94ea2c980d748c6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37365 } health_report { overall_health: UNKNOWN } } }
I20260927 14:55:34.386397 20586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:55:34.383065 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54496 (local address 127.0.98.3:40021)
0927 14:55:34.383205 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0927 14:55:34.383209 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 14:55:34.383221 (+ 12us) server_negotiation.cc:407] Connection header received
0927 14:55:34.383287 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:55:34.383289 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:55:34.383336 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:55:34.383423 (+ 87us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 14:55:34.383660 (+ 237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:55:34.384145 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:55:34.384983 (+ 838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:55:34.385253 (+ 270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:55:34.385333 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:55:34.385620 (+ 287us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 14:55:34.386013 (+ 393us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 14:55:34.386098 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0927 14:55:34.386171 (+ 73us) server_negotiation.cc:299] Negotiation successful
0927 14:55:34.386212 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260927 14:55:34.392076 20264 raft_consensus.cc:1261] T 9e852e63dd3d4c48955a98426d3b12ab P e66ebbf124d84cf1a94ea2c980d748c6 [term 1 FOLLOWER]: Refusing update from remote peer 1e5a252a48594ebeb458fad76c59dcb6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 14:55:34.392076 20395 raft_consensus.cc:1261] T 9e852e63dd3d4c48955a98426d3b12ab P cb7842c8ae564181a7185c06fb968d8d [term 1 FOLLOWER]: Refusing update from remote peer 1e5a252a48594ebeb458fad76c59dcb6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 14:55:34.392799 20597 consensus_queue.cc:1035] T 9e852e63dd3d4c48955a98426d3b12ab P 1e5a252a48594ebeb458fad76c59dcb6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e66ebbf124d84cf1a94ea2c980d748c6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37365 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 14:55:34.392975 20599 consensus_queue.cc:1035] T 9e852e63dd3d4c48955a98426d3b12ab P 1e5a252a48594ebeb458fad76c59dcb6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cb7842c8ae564181a7185c06fb968d8d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 14:55:34.400785 20603 mvcc.cc:205] Tried to move back new op lower bound from 7333973747262840832 to 7333973747151560704. Current Snapshot: MvccSnapshot[applied={T|T < 7333973747262840832}]
I20260927 14:55:34.403959 20604 mvcc.cc:205] Tried to move back new op lower bound from 7333973747262840832 to 7333973747151560704. Current Snapshot: MvccSnapshot[applied={T|T < 7333973747262840832 or (T in {7333973747262840832})}]
I20260927 14:55:34.409650 20121 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:50064:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 14:55:34.409806 20121 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:50064:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 14:55:34.412312 20121 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6532df0ca21842419e5b4a586eea70fb: Sending DeleteTablet for 3 replicas of tablet 9e852e63dd3d4c48955a98426d3b12ab
I20260927 14:55:34.413141 20375 tablet_service.cc:1514] Processing DeleteTablet for tablet 9e852e63dd3d4c48955a98426d3b12ab with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 14:55:34 UTC) from {username='slave'} at 127.0.0.1:43280
I20260927 14:55:34.413161 20506 tablet_service.cc:1514] Processing DeleteTablet for tablet 9e852e63dd3d4c48955a98426d3b12ab with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 14:55:34 UTC) from {username='slave'} at 127.0.0.1:54482
I20260927 14:55:34.413668 20613 tablet_replica.cc:330] T 9e852e63dd3d4c48955a98426d3b12ab P cb7842c8ae564181a7185c06fb968d8d: stopping tablet replica
I20260927 14:55:34.413705 20244 tablet_service.cc:1514] Processing DeleteTablet for tablet 9e852e63dd3d4c48955a98426d3b12ab with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 14:55:34 UTC) from {username='slave'} at 127.0.0.1:42820
I20260927 14:55:34.414011 20613 raft_consensus.cc:2229] T 9e852e63dd3d4c48955a98426d3b12ab P cb7842c8ae564181a7185c06fb968d8d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 14:55:34.414214 20614 tablet_replica.cc:330] T 9e852e63dd3d4c48955a98426d3b12ab P e66ebbf124d84cf1a94ea2c980d748c6: stopping tablet replica
I20260927 14:55:34.414355 20613 raft_consensus.cc:2258] T 9e852e63dd3d4c48955a98426d3b12ab P cb7842c8ae564181a7185c06fb968d8d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 14:55:34.414649 20614 raft_consensus.cc:2229] T 9e852e63dd3d4c48955a98426d3b12ab P e66ebbf124d84cf1a94ea2c980d748c6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 14:55:34.414749 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6936m5/build/debug/bin/kudu with pid 20182
I20260927 14:55:34.415027 20614 raft_consensus.cc:2258] T 9e852e63dd3d4c48955a98426d3b12ab P e66ebbf124d84cf1a94ea2c980d748c6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 14:55:34.416427 20613 ts_tablet_manager.cc:1905] T 9e852e63dd3d4c48955a98426d3b12ab P cb7842c8ae564181a7185c06fb968d8d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 14:55:34.419792 20612 tablet_replica.cc:330] T 9e852e63dd3d4c48955a98426d3b12ab P 1e5a252a48594ebeb458fad76c59dcb6: stopping tablet replica
I20260927 14:55:34.419912 20613 ts_tablet_manager.cc:1918] T 9e852e63dd3d4c48955a98426d3b12ab P cb7842c8ae564181a7185c06fb968d8d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 14:55:34.420018 20613 log.cc:1196] T 9e852e63dd3d4c48955a98426d3b12ab P cb7842c8ae564181a7185c06fb968d8d: Deleting WAL directory at /tmp/dist-test-task6936m5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790520684602251-392-0/minicluster-data/ts-1/wal/wals/9e852e63dd3d4c48955a98426d3b12ab
I20260927 14:55:34.420248 20612 raft_consensus.cc:2229] T 9e852e63dd3d4c48955a98426d3b12ab P 1e5a252a48594ebeb458fad76c59dcb6 [term 1 LEADER]: Raft consensus shutting down.
I20260927 14:55:34.420342 20613 ts_tablet_manager.cc:1939] T 9e852e63dd3d4c48955a98426d3b12ab P cb7842c8ae564181a7185c06fb968d8d: Deleting consensus metadata
I20260927 14:55:34.420769 20612 raft_consensus.cc:2258] T 9e852e63dd3d4c48955a98426d3b12ab P 1e5a252a48594ebeb458fad76c59dcb6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 14:55:34.421677 20106 catalog_manager.cc:4841] TS cb7842c8ae564181a7185c06fb968d8d (127.0.98.2:34445): tablet 9e852e63dd3d4c48955a98426d3b12ab (table test-table [id=aad43c09acfa466bb527603d4ce18351]) successfully deleted
I20260927 14:55:34.422354 20612 ts_tablet_manager.cc:1905] T 9e852e63dd3d4c48955a98426d3b12ab P 1e5a252a48594ebeb458fad76c59dcb6: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260927 14:55:34.423848 20106 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:37365 (error 108)
I20260927 14:55:34.424652 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6936m5/build/debug/bin/kudu with pid 20313
I20260927 14:55:34.424866 20585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:55:34.424567 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37365 (local address 127.0.0.1:42834)
0927 14:55:34.424702 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0927 14:55:34.424777 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:37365: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
I20260927 14:55:34.425562 20612 ts_tablet_manager.cc:1918] T 9e852e63dd3d4c48955a98426d3b12ab P 1e5a252a48594ebeb458fad76c59dcb6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 14:55:34.425642 20612 log.cc:1196] T 9e852e63dd3d4c48955a98426d3b12ab P 1e5a252a48594ebeb458fad76c59dcb6: Deleting WAL directory at /tmp/dist-test-task6936m5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790520684602251-392-0/minicluster-data/ts-2/wal/wals/9e852e63dd3d4c48955a98426d3b12ab
W20260927 14:55:34.425849 20106 catalog_manager.cc:4584] TS e66ebbf124d84cf1a94ea2c980d748c6 (127.0.98.1:37365): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 9e852e63dd3d4c48955a98426d3b12ab: Network error: Client connection negotiation failed: client connection to 127.0.98.1:37365: connect: Connection refused (error 111)
I20260927 14:55:34.425957 20612 ts_tablet_manager.cc:1939] T 9e852e63dd3d4c48955a98426d3b12ab P 1e5a252a48594ebeb458fad76c59dcb6: Deleting consensus metadata
I20260927 14:55:34.427052 20106 catalog_manager.cc:4841] TS 1e5a252a48594ebeb458fad76c59dcb6 (127.0.98.3:40021): tablet 9e852e63dd3d4c48955a98426d3b12ab (table test-table [id=aad43c09acfa466bb527603d4ce18351]) successfully deleted
I20260927 14:55:34.430987 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6936m5/build/debug/bin/kudu with pid 20444
I20260927 14:55:34.435926 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6936m5/build/debug/bin/kudu with pid 20091
2026-09-27T14:55:34Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (1135 ms)
[----------] 6 tests from EncryptionPolicyTest (10174 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-task6936m5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790520684602251-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 14:55:34 dist-test-slave-xk1g krb5kdc[20618](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 14:55:34 dist-test-slave-xk1g krb5kdc[20618](info): set up 2 sockets
Sep 27 14:55:34 dist-test-slave-xk1g krb5kdc[20618](info): commencing operation
krb5kdc: starting...
W20260927 14:55:35.497012 392 mini_kdc.cc:121] Time spent starting KDC: real 1.040s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6936m5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790520684602251-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6936m5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790520684602251-392-0/krb5kdc/test-user.keytab.
Sep 27 14:55:35 dist-test-slave-xk1g krb5kdc[20618](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790520935, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T14:55:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T14:55:35Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6936m5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790520684602251-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6936m5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790520684602251-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 14:55:35.616330 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6936m5/build/debug/bin/kudu
/tmp/dist-test-task6936m5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790520684602251-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790520684602251-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790520684602251-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790520684602251-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37009
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790520684602251-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45441
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37009
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790520684602251-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790520684602251-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790520684602251-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 14:55:35.722985 20631 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 14:55:35.723285 20631 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 14:55:35.723400 20631 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 14:55:35.727473 20631 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 14:55:35.727567 20631 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 14:55:35.727607 20631 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 14:55:35.727664 20631 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 14:55:35.727702 20631 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 14:55:35.732914 20631 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45441
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790520684602251-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790520684602251-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37009
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790520684602251-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37009
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790520684602251-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20631
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790520684602251-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:15:17 UTC on ccycloud.quasar-ledxfe.root.comops.site
build id 4676134
I20260927 14:55:35.733968 20631 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 14:55:35.734957 20631 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 14:55:35.743351 20637 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 14:55:35.743419 20636 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 14:55:35.743351 20639 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 14:55:35.743672 20631 server_base.cc:1034] running on GCE node
I20260927 14:55:35.744237 20631 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 14:55:35.745404 20631 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 14:55:35.746589 20631 hybrid_clock.cc:648] HybridClock initialized: now 1790520935746577 us; error 25 us; skew 500 ppm
Sep 27 14:55:35 dist-test-slave-xk1g krb5kdc[20618](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790520935, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 14:55:35.749264 20631 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 14:55:35.750120 20631 webserver.cc:460] Webserver started at http://127.0.0.1:42105/ using document root <none> and password file <none>
I20260927 14:55:35.750625 20631 fs_manager.cc:359] Metadata directory not provided
I20260927 14:55:35.750681 20631 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 14:55:35.750836 20631 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 14:55:35.752751 20631 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6936m5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790520684602251-392-0/minicluster-data/master-0/data/instance:
uuid: "6728ebbe3aca452aa95abe891e099262"
format_stamp: "Formatted at 2026-09-27 14:55:35 on dist-test-slave-xk1g"
server_key: "7513b78c66917d9f81c5fca4a3f5e537"
server_key_iv: "c54b9e915311c02027970da1c9e9aef2"
server_key_version: "encryptionkey@0"
I20260927 14:55:35.753307 20631 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6936m5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790520684602251-392-0/minicluster-data/master-0/wal/instance:
uuid: "6728ebbe3aca452aa95abe891e099262"
format_stamp: "Formatted at 2026-09-27 14:55:35 on dist-test-slave-xk1g"
server_key: "7513b78c66917d9f81c5fca4a3f5e537"
server_key_iv: "c54b9e915311c02027970da1c9e9aef2"
server_key_version: "encryptionkey@0"
I20260927 14:55:35.756673 20631 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260927 14:55:35.758948 20646 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 14:55:35.760210 20631 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 14:55:35.760310 20631 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790520684602251-392-0/minicluster-data/master-0/data,/tmp/dist-test-task6936m5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790520684602251-392-0/minicluster-data/master-0/wal
uuid: "6728ebbe3aca452aa95abe891e099262"
format_stamp: "Formatted at 2026-09-27 14:55:35 on dist-test-slave-xk1g"
server_key: "7513b78c66917d9f81c5fca4a3f5e537"
server_key_iv: "c54b9e915311c02027970da1c9e9aef2"
server_key_version: "encryptionkey@0"
I20260927 14:55:35.760392 20631 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790520684602251-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790520684602251-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790520684602251-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 14:55:35.776703 20631 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 14:55:35.779397 20631 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 14:55:35.779542 20631 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 14:55:35.786568 20631 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37009
I20260927 14:55:35.786585 20698 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37009 every 8 connection(s)
I20260927 14:55:35.787560 20631 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6936m5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790520684602251-392-0/minicluster-data/master-0/data/info.pb
I20260927 14:55:35.790262 20699 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 14:55:35.792081 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6936m5/build/debug/bin/kudu as pid 20631
I20260927 14:55:35.792240 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6936m5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790520684602251-392-0/minicluster-data/master-0/wal/instance
I20260927 14:55:35.792500 392 external_mini_cluster.cc:1428] Setting key 5f399da64cbb57b5abefd68e89dfcf1d
I20260927 14:55:35.796573 20699 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6728ebbe3aca452aa95abe891e099262: Bootstrap starting.
I20260927 14:55:35.799081 20699 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6728ebbe3aca452aa95abe891e099262: Neither blocks nor log segments found. Creating new log.
I20260927 14:55:35.799873 20699 log.cc:824] T 00000000000000000000000000000000 P 6728ebbe3aca452aa95abe891e099262: Log is configured to *not* fsync() on all Append() calls
I20260927 14:55:35.801831 20699 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6728ebbe3aca452aa95abe891e099262: No bootstrap required, opened a new log
Sep 27 14:55:35 dist-test-slave-xk1g krb5kdc[20618](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790520935, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260927 14:55:35.805433 20699 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6728ebbe3aca452aa95abe891e099262 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6728ebbe3aca452aa95abe891e099262" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37009 } }
I20260927 14:55:35.805701 20699 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6728ebbe3aca452aa95abe891e099262 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 14:55:35.805759 20699 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6728ebbe3aca452aa95abe891e099262 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6728ebbe3aca452aa95abe891e099262, State: Initialized, Role: FOLLOWER
I20260927 14:55:35.806234 20699 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6728ebbe3aca452aa95abe891e099262 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6728ebbe3aca452aa95abe891e099262" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37009 } }
I20260927 14:55:35.806402 20699 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6728ebbe3aca452aa95abe891e099262 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 14:55:35.806488 20699 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6728ebbe3aca452aa95abe891e099262 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 14:55:35.806607 20699 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6728ebbe3aca452aa95abe891e099262 [term 0 FOLLOWER]: Advancing to term 1
I20260927 14:55:35.807652 20699 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6728ebbe3aca452aa95abe891e099262 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6728ebbe3aca452aa95abe891e099262" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37009 } }
I20260927 14:55:35.807986 20699 leader_election.cc:302] T 00000000000000000000000000000000 P 6728ebbe3aca452aa95abe891e099262 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6728ebbe3aca452aa95abe891e099262; no voters:
I20260927 14:55:35.808291 20699 leader_election.cc:288] T 00000000000000000000000000000000 P 6728ebbe3aca452aa95abe891e099262 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 14:55:35.808393 20704 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6728ebbe3aca452aa95abe891e099262 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 14:55:35.808499 20702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:55:35.794038 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33915 (local address 127.0.0.1:37009)
0927 14:55:35.794464 (+ 426us) server_negotiation.cc:206] Beginning negotiation
0927 14:55:35.794484 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0927 14:55:35.794526 (+ 42us) server_negotiation.cc:407] Connection header received
0927 14:55:35.795535 (+ 1009us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:55:35.795558 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:55:35.795981 (+ 423us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:55:35.796329 (+ 348us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:55:35.797084 (+ 755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:55:35.798527 (+ 1443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:55:35.799566 (+ 1039us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:55:35.800272 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:55:35.800445 (+ 173us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:55:35.803207 (+ 2762us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:55:35.803213 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:55:35.803224 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 14:55:35.803249 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:55:35.805454 (+ 2205us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:55:35.806003 (+ 549us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:55:35.806012 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:55:35.806018 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:55:35.806144 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:55:35.806496 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:55:35.806500 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:55:35.806502 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:55:35.806952 (+ 450us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:55:35.807121 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0927 14:55:35.807417 (+ 296us) server_negotiation.cc:299] Negotiation successful
0927 14:55:35.807634 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":282,"thread_start_us":113,"threads_started":1}
I20260927 14:55:35.808593 20704 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6728ebbe3aca452aa95abe891e099262 [term 1 LEADER]: Becoming Leader. State: Replica: 6728ebbe3aca452aa95abe891e099262, State: Running, Role: LEADER
I20260927 14:55:35.808876 20704 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6728ebbe3aca452aa95abe891e099262 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6728ebbe3aca452aa95abe891e099262" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37009 } }
I20260927 14:55:35.809602 20699 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6728ebbe3aca452aa95abe891e099262 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 14:55:35.810096 20706 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6728ebbe3aca452aa95abe891e099262 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6728ebbe3aca452aa95abe891e099262. Latest consensus state: current_term: 1 leader_uuid: "6728ebbe3aca452aa95abe891e099262" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6728ebbe3aca452aa95abe891e099262" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37009 } } }
I20260927 14:55:35.810215 20706 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6728ebbe3aca452aa95abe891e099262 [sys.catalog]: This master's current role is: LEADER
I20260927 14:55:35.810510 20705 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6728ebbe3aca452aa95abe891e099262 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6728ebbe3aca452aa95abe891e099262" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6728ebbe3aca452aa95abe891e099262" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37009 } } }
I20260927 14:55:35.810597 20705 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6728ebbe3aca452aa95abe891e099262 [sys.catalog]: This master's current role is: LEADER
I20260927 14:55:35.810933 20710 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 14:55:35.811998 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 14:55:35.813601 20710 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 14:55:35.819490 20710 catalog_manager.cc:1347] Generated new cluster ID: 917e7b4d3a5b40f69c33d37c445da21f
I20260927 14:55:35.819562 20710 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 14:55:35.823371 20702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:55:35.814523 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44007 (local address 127.0.0.1:37009)
0927 14:55:35.814652 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0927 14:55:35.814657 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 14:55:35.814904 (+ 247us) server_negotiation.cc:407] Connection header received
0927 14:55:35.815025 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:55:35.815029 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:55:35.815076 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:55:35.815160 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:55:35.815917 (+ 757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:55:35.816691 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:55:35.817746 (+ 1055us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:55:35.817992 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:55:35.818100 (+ 108us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:55:35.819726 (+ 1626us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:55:35.819729 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:55:35.819732 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 14:55:35.819756 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:55:35.821433 (+ 1677us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:55:35.821937 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:55:35.821941 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:55:35.821942 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:55:35.822015 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:55:35.822350 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:55:35.822353 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:55:35.822355 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:55:35.822571 (+ 216us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:55:35.822754 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0927 14:55:35.823131 (+ 377us) server_negotiation.cc:299] Negotiation successful
0927 14:55:35.823224 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260927 14:55:35.843483 20710 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 14:55:35.844769 20710 catalog_manager.cc:1504] Loading token signing keys...
I20260927 14:55:35.857549 20710 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6728ebbe3aca452aa95abe891e099262: Generated new TSK 0
I20260927 14:55:35.858274 20710 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 14:55:35.866530 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6936m5/build/debug/bin/kudu with pid 20631
W20260927 14:55:35.872099 20616 connection.cc:535] client connection to 127.0.0.1:37009 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:37009): Connection reset by peer (error 104)
2026-09-27T14:55:35Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (1429 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-task6936m5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790520684602251-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 14:55:35 dist-test-slave-xk1g krb5kdc[20730](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 14:55:35 dist-test-slave-xk1g krb5kdc[20730](info): set up 2 sockets
Sep 27 14:55:35 dist-test-slave-xk1g krb5kdc[20730](info): commencing operation
krb5kdc: starting...
W20260927 14:55:36.921283 392 mini_kdc.cc:121] Time spent starting KDC: real 1.035s user 0.002s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6936m5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790520684602251-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6936m5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790520684602251-392-0/krb5kdc/test-user.keytab.
Sep 27 14:55:37 dist-test-slave-xk1g krb5kdc[20730](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790520937, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T14:55:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T14:55:37Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6936m5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790520684602251-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6936m5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790520684602251-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 14:55:37.046191 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6936m5/build/debug/bin/kudu
/tmp/dist-test-task6936m5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790520684602251-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790520684602251-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790520684602251-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790520684602251-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41599
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790520684602251-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40323
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41599
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790520684602251-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790520684602251-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790520684602251-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 14:55:37.154210 20743 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 14:55:37.154516 20743 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 14:55:37.154583 20743 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 14:55:37.158567 20743 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 14:55:37.158658 20743 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 14:55:37.158694 20743 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 14:55:37.158730 20743 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 14:55:37.158762 20743 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 14:55:37.163940 20743 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40323
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790520684602251-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790520684602251-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41599
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790520684602251-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41599
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790520684602251-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20743
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790520684602251-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:15:17 UTC on ccycloud.quasar-ledxfe.root.comops.site
build id 4676134
I20260927 14:55:37.165016 20743 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 14:55:37.166078 20743 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 14:55:37.174190 20751 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 14:55:37.174185 20748 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 14:55:37.174199 20749 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 14:55:37.174338 20743 server_base.cc:1034] running on GCE node
I20260927 14:55:37.174984 20743 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 14:55:37.175984 20743 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 14:55:37.177126 20743 hybrid_clock.cc:648] HybridClock initialized: now 1790520937177095 us; error 42 us; skew 500 ppm
Sep 27 14:55:37 dist-test-slave-xk1g krb5kdc[20730](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790520937, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 14:55:37.179821 20743 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 14:55:37.180683 20743 webserver.cc:460] Webserver started at http://127.0.0.1:39415/ using document root <none> and password file <none>
I20260927 14:55:37.181317 20743 fs_manager.cc:359] Metadata directory not provided
I20260927 14:55:37.181378 20743 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 14:55:37.181608 20743 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 14:55:37.183573 20743 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6936m5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790520684602251-392-0/minicluster-data/master-0/data/instance:
uuid: "30502dc27c9b48449e317a2819c8bfd0"
format_stamp: "Formatted at 2026-09-27 14:55:37 on dist-test-slave-xk1g"
server_key: "bc244a13989202b6895864ca13f2af21"
server_key_iv: "0758c4d6022025ddfc1ac6abe47e18bd"
server_key_version: "encryptionkey@0"
I20260927 14:55:37.184135 20743 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6936m5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790520684602251-392-0/minicluster-data/master-0/wal/instance:
uuid: "30502dc27c9b48449e317a2819c8bfd0"
format_stamp: "Formatted at 2026-09-27 14:55:37 on dist-test-slave-xk1g"
server_key: "bc244a13989202b6895864ca13f2af21"
server_key_iv: "0758c4d6022025ddfc1ac6abe47e18bd"
server_key_version: "encryptionkey@0"
I20260927 14:55:37.187764 20743 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260927 14:55:37.190057 20758 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 14:55:37.191162 20743 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 14:55:37.191293 20743 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790520684602251-392-0/minicluster-data/master-0/data,/tmp/dist-test-task6936m5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790520684602251-392-0/minicluster-data/master-0/wal
uuid: "30502dc27c9b48449e317a2819c8bfd0"
format_stamp: "Formatted at 2026-09-27 14:55:37 on dist-test-slave-xk1g"
server_key: "bc244a13989202b6895864ca13f2af21"
server_key_iv: "0758c4d6022025ddfc1ac6abe47e18bd"
server_key_version: "encryptionkey@0"
I20260927 14:55:37.191428 20743 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790520684602251-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790520684602251-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790520684602251-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 14:55:37.199913 20743 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 14:55:37.202788 20743 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 14:55:37.202993 20743 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 14:55:37.209821 20743 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41599
I20260927 14:55:37.209836 20810 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41599 every 8 connection(s)
I20260927 14:55:37.210736 20743 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6936m5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790520684602251-392-0/minicluster-data/master-0/data/info.pb
I20260927 14:55:37.211707 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6936m5/build/debug/bin/kudu as pid 20743
I20260927 14:55:37.211850 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6936m5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790520684602251-392-0/minicluster-data/master-0/wal/instance
I20260927 14:55:37.212122 392 external_mini_cluster.cc:1428] Setting key 960e6039b2b8289ca3724ee039d8850b
I20260927 14:55:37.213693 20811 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 14:55:37.219115 20811 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 30502dc27c9b48449e317a2819c8bfd0: Bootstrap starting.
I20260927 14:55:37.220901 20814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:55:37.213500 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56995 (local address 127.0.0.1:41599)
0927 14:55:37.213951 (+ 451us) server_negotiation.cc:206] Beginning negotiation
0927 14:55:37.213966 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 14:55:37.214000 (+ 34us) server_negotiation.cc:407] Connection header received
0927 14:55:37.214622 (+ 622us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:55:37.214636 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:55:37.214963 (+ 327us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:55:37.215286 (+ 323us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:55:37.215678 (+ 392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:55:37.216653 (+ 975us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:55:37.217614 (+ 961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:55:37.218082 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:55:37.218250 (+ 168us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:55:37.219167 (+ 917us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:55:37.219172 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:55:37.219204 (+ 32us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 14:55:37.219606 (+ 402us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:55:37.219659 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 14:55:37.219681 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:55:37.219824 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0927 14:55:37.220032 (+ 208us) server_negotiation.cc:299] Negotiation successful
0927 14:55:37.220167 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":308,"thread_start_us":146,"threads_started":1}
I20260927 14:55:37.221490 20811 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 30502dc27c9b48449e317a2819c8bfd0: Neither blocks nor log segments found. Creating new log.
I20260927 14:55:37.222324 20811 log.cc:824] T 00000000000000000000000000000000 P 30502dc27c9b48449e317a2819c8bfd0: Log is configured to *not* fsync() on all Append() calls
I20260927 14:55:37.223819 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 14:55:37.224366 20811 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 30502dc27c9b48449e317a2819c8bfd0: No bootstrap required, opened a new log
I20260927 14:55:37.228673 20811 raft_consensus.cc:348] T 00000000000000000000000000000000 P 30502dc27c9b48449e317a2819c8bfd0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30502dc27c9b48449e317a2819c8bfd0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41599 } }
I20260927 14:55:37.228932 20811 raft_consensus.cc:374] T 00000000000000000000000000000000 P 30502dc27c9b48449e317a2819c8bfd0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 14:55:37.229018 20811 raft_consensus.cc:729] T 00000000000000000000000000000000 P 30502dc27c9b48449e317a2819c8bfd0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30502dc27c9b48449e317a2819c8bfd0, State: Initialized, Role: FOLLOWER
I20260927 14:55:37.229544 20811 consensus_queue.cc:260] T 00000000000000000000000000000000 P 30502dc27c9b48449e317a2819c8bfd0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30502dc27c9b48449e317a2819c8bfd0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41599 } }
I20260927 14:55:37.229709 20811 raft_consensus.cc:388] T 00000000000000000000000000000000 P 30502dc27c9b48449e317a2819c8bfd0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 14:55:37.229780 20811 raft_consensus.cc:482] T 00000000000000000000000000000000 P 30502dc27c9b48449e317a2819c8bfd0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 14:55:37.229883 20811 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 30502dc27c9b48449e317a2819c8bfd0 [term 0 FOLLOWER]: Advancing to term 1
I20260927 14:55:37.230070 20814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:55:37.225746 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38973 (local address 127.0.0.1:41599)
0927 14:55:37.225877 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0927 14:55:37.225881 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 14:55:37.226130 (+ 249us) server_negotiation.cc:407] Connection header received
0927 14:55:37.226256 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:55:37.226262 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:55:37.226314 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:55:37.226390 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:55:37.226641 (+ 251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:55:37.227457 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:55:37.228246 (+ 789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:55:37.228573 (+ 327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:55:37.228681 (+ 108us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:55:37.229576 (+ 895us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:55:37.229580 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:55:37.229582 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 14:55:37.229594 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:55:37.229620 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 14:55:37.229633 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:55:37.229739 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0927 14:55:37.229898 (+ 159us) server_negotiation.cc:299] Negotiation successful
0927 14:55:37.229937 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260927 14:55:37.230921 20811 raft_consensus.cc:504] T 00000000000000000000000000000000 P 30502dc27c9b48449e317a2819c8bfd0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30502dc27c9b48449e317a2819c8bfd0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41599 } }
I20260927 14:55:37.231212 20811 leader_election.cc:302] T 00000000000000000000000000000000 P 30502dc27c9b48449e317a2819c8bfd0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 30502dc27c9b48449e317a2819c8bfd0; no voters:
I20260927 14:55:37.231446 20811 leader_election.cc:288] T 00000000000000000000000000000000 P 30502dc27c9b48449e317a2819c8bfd0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 14:55:37.231565 20821 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 30502dc27c9b48449e317a2819c8bfd0 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 14:55:37.231839 20821 raft_consensus.cc:686] T 00000000000000000000000000000000 P 30502dc27c9b48449e317a2819c8bfd0 [term 1 LEADER]: Becoming Leader. State: Replica: 30502dc27c9b48449e317a2819c8bfd0, State: Running, Role: LEADER
I20260927 14:55:37.232156 20821 consensus_queue.cc:237] T 00000000000000000000000000000000 P 30502dc27c9b48449e317a2819c8bfd0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30502dc27c9b48449e317a2819c8bfd0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41599 } }
I20260927 14:55:37.232622 20811 sys_catalog.cc:564] T 00000000000000000000000000000000 P 30502dc27c9b48449e317a2819c8bfd0 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 14:55:37.233819 20822 sys_catalog.cc:455] T 00000000000000000000000000000000 P 30502dc27c9b48449e317a2819c8bfd0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "30502dc27c9b48449e317a2819c8bfd0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30502dc27c9b48449e317a2819c8bfd0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41599 } } }
I20260927 14:55:37.233945 20822 sys_catalog.cc:458] T 00000000000000000000000000000000 P 30502dc27c9b48449e317a2819c8bfd0 [sys.catalog]: This master's current role is: LEADER
I20260927 14:55:37.234228 20823 sys_catalog.cc:455] T 00000000000000000000000000000000 P 30502dc27c9b48449e317a2819c8bfd0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 30502dc27c9b48449e317a2819c8bfd0. Latest consensus state: current_term: 1 leader_uuid: "30502dc27c9b48449e317a2819c8bfd0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30502dc27c9b48449e317a2819c8bfd0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41599 } } }
I20260927 14:55:37.234324 20823 sys_catalog.cc:458] T 00000000000000000000000000000000 P 30502dc27c9b48449e317a2819c8bfd0 [sys.catalog]: This master's current role is: LEADER
I20260927 14:55:37.234400 20831 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 14:55:37.236997 20831 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 14:55:37.242211 20831 catalog_manager.cc:1347] Generated new cluster ID: ae78f3e078034da3a16262a41010bccc
I20260927 14:55:37.242280 20831 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 14:55:37.252779 20831 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 14:55:37.253965 20831 catalog_manager.cc:1504] Loading token signing keys...
I20260927 14:55:37.268213 20831 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 30502dc27c9b48449e317a2819c8bfd0: Generated new TSK 0
I20260927 14:55:37.269037 20831 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 14:55:37.280392 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6936m5/build/debug/bin/kudu with pid 20743
W20260927 14:55:37.285679 20728 connection.cc:535] client connection to 127.0.0.1:41599 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:41599): Connection reset by peer (error 104)
2026-09-27T14:55:37Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1413 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-task6936m5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790520684602251-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 14:55:37 dist-test-slave-xk1g krb5kdc[20842](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 14:55:37 dist-test-slave-xk1g krb5kdc[20842](info): set up 2 sockets
Sep 27 14:55:37 dist-test-slave-xk1g krb5kdc[20842](info): commencing operation
krb5kdc: starting...
W20260927 14:55:38.309377 392 mini_kdc.cc:121] Time spent starting KDC: real 1.009s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6936m5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790520684602251-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6936m5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790520684602251-392-0/krb5kdc/test-user.keytab.
Sep 27 14:55:38 dist-test-slave-xk1g krb5kdc[20842](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790520938, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T14:55:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T14:55:38Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6936m5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790520684602251-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6936m5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790520684602251-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 14:55:38.429659 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6936m5/build/debug/bin/kudu
/tmp/dist-test-task6936m5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790520684602251-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790520684602251-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790520684602251-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790520684602251-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44567
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790520684602251-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39785
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44567
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790520684602251-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790520684602251-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790520684602251-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 14:55:38.539384 20855 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 14:55:38.539709 20855 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 14:55:38.539796 20855 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 14:55:38.543962 20855 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 14:55:38.544063 20855 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 14:55:38.544100 20855 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 14:55:38.544165 20855 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 14:55:38.544203 20855 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 14:55:38.549827 20855 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39785
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790520684602251-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790520684602251-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44567
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790520684602251-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44567
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790520684602251-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20855
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790520684602251-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:15:17 UTC on ccycloud.quasar-ledxfe.root.comops.site
build id 4676134
I20260927 14:55:38.551237 20855 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 14:55:38.552671 20855 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 14:55:38.561110 20861 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 14:55:38.561127 20860 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 14:55:38.561127 20863 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 14:55:38.561398 20855 server_base.cc:1034] running on GCE node
I20260927 14:55:38.562011 20855 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 14:55:38.563122 20855 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 14:55:38.564329 20855 hybrid_clock.cc:648] HybridClock initialized: now 1790520938564313 us; error 36 us; skew 500 ppm
Sep 27 14:55:38 dist-test-slave-xk1g krb5kdc[20842](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790520938, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 14:55:38.567234 20855 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 14:55:38.568120 20855 webserver.cc:460] Webserver started at http://127.0.0.1:40985/ using document root <none> and password file <none>
I20260927 14:55:38.568640 20855 fs_manager.cc:359] Metadata directory not provided
I20260927 14:55:38.568697 20855 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 14:55:38.568859 20855 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 14:55:38.570901 20855 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6936m5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790520684602251-392-0/minicluster-data/master-0/data/instance:
uuid: "6dbac19f18464c95a7f5ccfd8493dea7"
format_stamp: "Formatted at 2026-09-27 14:55:38 on dist-test-slave-xk1g"
server_key: "7dedb566335d39727fbd85362244bc6a"
server_key_iv: "1fa9b7f67ea060530871fec2ed9e1b36"
server_key_version: "encryptionkey@0"
I20260927 14:55:38.571410 20855 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6936m5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790520684602251-392-0/minicluster-data/master-0/wal/instance:
uuid: "6dbac19f18464c95a7f5ccfd8493dea7"
format_stamp: "Formatted at 2026-09-27 14:55:38 on dist-test-slave-xk1g"
server_key: "7dedb566335d39727fbd85362244bc6a"
server_key_iv: "1fa9b7f67ea060530871fec2ed9e1b36"
server_key_version: "encryptionkey@0"
I20260927 14:55:38.574976 20855 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260927 14:55:38.577298 20870 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 14:55:38.578507 20855 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 14:55:38.578634 20855 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790520684602251-392-0/minicluster-data/master-0/data,/tmp/dist-test-task6936m5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790520684602251-392-0/minicluster-data/master-0/wal
uuid: "6dbac19f18464c95a7f5ccfd8493dea7"
format_stamp: "Formatted at 2026-09-27 14:55:38 on dist-test-slave-xk1g"
server_key: "7dedb566335d39727fbd85362244bc6a"
server_key_iv: "1fa9b7f67ea060530871fec2ed9e1b36"
server_key_version: "encryptionkey@0"
I20260927 14:55:38.578747 20855 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790520684602251-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790520684602251-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790520684602251-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 14:55:38.583298 20855 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 14:55:38.590004 20855 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 14:55:38.590215 20855 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 14:55:38.597263 20855 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44567
I20260927 14:55:38.597278 20922 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44567 every 8 connection(s)
I20260927 14:55:38.598174 20855 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6936m5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790520684602251-392-0/minicluster-data/master-0/data/info.pb
I20260927 14:55:38.600822 20923 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 14:55:38.606035 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6936m5/build/debug/bin/kudu as pid 20855
I20260927 14:55:38.606194 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6936m5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790520684602251-392-0/minicluster-data/master-0/wal/instance
I20260927 14:55:38.606446 392 external_mini_cluster.cc:1428] Setting key 57c79f4c197713585597af1c086e9640
I20260927 14:55:38.606468 20923 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6dbac19f18464c95a7f5ccfd8493dea7: Bootstrap starting.
I20260927 14:55:38.609014 20923 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6dbac19f18464c95a7f5ccfd8493dea7: Neither blocks nor log segments found. Creating new log.
I20260927 14:55:38.609781 20923 log.cc:824] T 00000000000000000000000000000000 P 6dbac19f18464c95a7f5ccfd8493dea7: Log is configured to *not* fsync() on all Append() calls
I20260927 14:55:38.611681 20926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:55:38.607849 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59093 (local address 127.0.0.1:44567)
0927 14:55:38.608318 (+ 469us) server_negotiation.cc:206] Beginning negotiation
0927 14:55:38.608332 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 14:55:38.608370 (+ 38us) server_negotiation.cc:407] Connection header received
0927 14:55:38.608993 (+ 623us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:55:38.609010 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:55:38.609416 (+ 406us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:55:38.609762 (+ 346us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:55:38.610524 (+ 762us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:55:38.610529 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:55:38.610541 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 14:55:38.610551 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:55:38.610602 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 14:55:38.610620 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:55:38.610741 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0927 14:55:38.610851 (+ 110us) server_negotiation.cc:299] Negotiation successful
0927 14:55:38.611012 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":303,"thread_start_us":138,"threads_started":1}
I20260927 14:55:38.611703 20923 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6dbac19f18464c95a7f5ccfd8493dea7: No bootstrap required, opened a new log
I20260927 14:55:38.614349 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 14:55:38.615504 20923 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6dbac19f18464c95a7f5ccfd8493dea7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6dbac19f18464c95a7f5ccfd8493dea7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44567 } }
I20260927 14:55:38.615804 20923 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6dbac19f18464c95a7f5ccfd8493dea7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 14:55:38.615887 20923 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6dbac19f18464c95a7f5ccfd8493dea7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6dbac19f18464c95a7f5ccfd8493dea7, State: Initialized, Role: FOLLOWER
I20260927 14:55:38.616380 20923 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6dbac19f18464c95a7f5ccfd8493dea7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6dbac19f18464c95a7f5ccfd8493dea7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44567 } }
I20260927 14:55:38.616555 20923 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6dbac19f18464c95a7f5ccfd8493dea7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 14:55:38.616631 20923 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6dbac19f18464c95a7f5ccfd8493dea7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 14:55:38.616748 20923 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6dbac19f18464c95a7f5ccfd8493dea7 [term 0 FOLLOWER]: Advancing to term 1
I20260927 14:55:38.618018 20926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:55:38.616369 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40289 (local address 127.0.0.1:44567)
0927 14:55:38.616476 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0927 14:55:38.616480 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 14:55:38.616687 (+ 207us) server_negotiation.cc:407] Connection header received
0927 14:55:38.616831 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:55:38.616835 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:55:38.616874 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:55:38.616951 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:55:38.617628 (+ 677us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:55:38.617630 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:55:38.617632 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 14:55:38.617642 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:55:38.617665 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 14:55:38.617675 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:55:38.617756 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0927 14:55:38.617872 (+ 116us) server_negotiation.cc:299] Negotiation successful
0927 14:55:38.617901 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260927 14:55:38.618055 20923 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6dbac19f18464c95a7f5ccfd8493dea7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6dbac19f18464c95a7f5ccfd8493dea7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44567 } }
I20260927 14:55:38.618425 20923 leader_election.cc:302] T 00000000000000000000000000000000 P 6dbac19f18464c95a7f5ccfd8493dea7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6dbac19f18464c95a7f5ccfd8493dea7; no voters:
I20260927 14:55:38.618785 20923 leader_election.cc:288] T 00000000000000000000000000000000 P 6dbac19f18464c95a7f5ccfd8493dea7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 14:55:38.618860 20933 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6dbac19f18464c95a7f5ccfd8493dea7 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 14:55:38.619086 20933 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6dbac19f18464c95a7f5ccfd8493dea7 [term 1 LEADER]: Becoming Leader. State: Replica: 6dbac19f18464c95a7f5ccfd8493dea7, State: Running, Role: LEADER
I20260927 14:55:38.619400 20933 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6dbac19f18464c95a7f5ccfd8493dea7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6dbac19f18464c95a7f5ccfd8493dea7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44567 } }
I20260927 14:55:38.620090 20923 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6dbac19f18464c95a7f5ccfd8493dea7 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 14:55:38.621222 20935 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6dbac19f18464c95a7f5ccfd8493dea7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6dbac19f18464c95a7f5ccfd8493dea7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6dbac19f18464c95a7f5ccfd8493dea7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44567 } } }
I20260927 14:55:38.621232 20934 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6dbac19f18464c95a7f5ccfd8493dea7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6dbac19f18464c95a7f5ccfd8493dea7. Latest consensus state: current_term: 1 leader_uuid: "6dbac19f18464c95a7f5ccfd8493dea7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6dbac19f18464c95a7f5ccfd8493dea7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44567 } } }
I20260927 14:55:38.621371 20934 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6dbac19f18464c95a7f5ccfd8493dea7 [sys.catalog]: This master's current role is: LEADER
I20260927 14:55:38.621371 20935 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6dbac19f18464c95a7f5ccfd8493dea7 [sys.catalog]: This master's current role is: LEADER
I20260927 14:55:38.621757 20942 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 14:55:38.624912 20942 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 14:55:38.630440 20942 catalog_manager.cc:1347] Generated new cluster ID: 8dd94f4181194121bac5ef628e178ed0
I20260927 14:55:38.630515 20942 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 14:55:38.639528 20942 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 14:55:38.639829 20942 catalog_manager.cc:1504] Loading token signing keys...
I20260927 14:55:38.648226 20942 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6dbac19f18464c95a7f5ccfd8493dea7: Generated new TSK 0
I20260927 14:55:38.648999 20942 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 14:55:38.657727 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6936m5/build/debug/bin/kudu with pid 20855
2026-09-27T14:55:38Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1376 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-task6936m5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790520684602251-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 14:55:38 dist-test-slave-xk1g krb5kdc[20954](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 14:55:38 dist-test-slave-xk1g krb5kdc[20954](info): set up 2 sockets
Sep 27 14:55:38 dist-test-slave-xk1g krb5kdc[20954](info): commencing operation
krb5kdc: starting...
W20260927 14:55:39.701364 392 mini_kdc.cc:121] Time spent starting KDC: real 1.024s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6936m5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790520684602251-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6936m5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790520684602251-392-0/krb5kdc/test-user.keytab.
Sep 27 14:55:39 dist-test-slave-xk1g krb5kdc[20954](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790520939, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T14:55:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T14:55:39Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6936m5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790520684602251-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6936m5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790520684602251-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 14:55:39.819434 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6936m5/build/debug/bin/kudu
/tmp/dist-test-task6936m5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790520684602251-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790520684602251-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790520684602251-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790520684602251-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37547
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790520684602251-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34177
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37547
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790520684602251-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790520684602251-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790520684602251-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 14:55:39.929919 20967 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 14:55:39.930141 20967 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 14:55:39.930179 20967 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 14:55:39.934267 20967 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 14:55:39.934320 20967 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 14:55:39.934340 20967 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 14:55:39.934355 20967 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 14:55:39.934369 20967 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 14:55:39.939416 20967 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34177
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790520684602251-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790520684602251-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37547
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790520684602251-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37547
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790520684602251-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20967
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790520684602251-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:15:17 UTC on ccycloud.quasar-ledxfe.root.comops.site
build id 4676134
I20260927 14:55:39.940356 20967 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 14:55:39.941470 20967 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 14:55:39.949788 20973 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 14:55:39.949842 20967 server_base.cc:1034] running on GCE node
W20260927 14:55:39.949740 20975 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 14:55:39.949739 20972 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 14:55:39.950701 20967 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 14:55:39.951737 20967 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 14:55:39.952925 20967 hybrid_clock.cc:648] HybridClock initialized: now 1790520939952901 us; error 29 us; skew 500 ppm
Sep 27 14:55:39 dist-test-slave-xk1g krb5kdc[20954](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790520939, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 14:55:39.959285 20967 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 14:55:39.960114 20967 webserver.cc:460] Webserver started at http://127.0.0.1:32875/ using document root <none> and password file <none>
I20260927 14:55:39.960642 20967 fs_manager.cc:359] Metadata directory not provided
I20260927 14:55:39.960702 20967 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 14:55:39.960858 20967 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 14:55:39.962885 20967 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6936m5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790520684602251-392-0/minicluster-data/master-0/data/instance:
uuid: "8241dc76c53648ea8f91ab03993f7045"
format_stamp: "Formatted at 2026-09-27 14:55:39 on dist-test-slave-xk1g"
server_key: "2df57fcf3f92f86d6e49903156f49954"
server_key_iv: "c14590761afd1ea63b036275ac0d1a66"
server_key_version: "encryptionkey@0"
I20260927 14:55:39.963392 20967 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6936m5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790520684602251-392-0/minicluster-data/master-0/wal/instance:
uuid: "8241dc76c53648ea8f91ab03993f7045"
format_stamp: "Formatted at 2026-09-27 14:55:39 on dist-test-slave-xk1g"
server_key: "2df57fcf3f92f86d6e49903156f49954"
server_key_iv: "c14590761afd1ea63b036275ac0d1a66"
server_key_version: "encryptionkey@0"
I20260927 14:55:39.966975 20967 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260927 14:55:39.969264 20982 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 14:55:39.970482 20967 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 14:55:39.970613 20967 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790520684602251-392-0/minicluster-data/master-0/data,/tmp/dist-test-task6936m5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790520684602251-392-0/minicluster-data/master-0/wal
uuid: "8241dc76c53648ea8f91ab03993f7045"
format_stamp: "Formatted at 2026-09-27 14:55:39 on dist-test-slave-xk1g"
server_key: "2df57fcf3f92f86d6e49903156f49954"
server_key_iv: "c14590761afd1ea63b036275ac0d1a66"
server_key_version: "encryptionkey@0"
I20260927 14:55:39.970726 20967 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790520684602251-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790520684602251-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790520684602251-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 14:55:40.008459 20967 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 14:55:40.011379 20967 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 14:55:40.011583 20967 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 14:55:40.018400 20967 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37547
I20260927 14:55:40.018424 21034 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37547 every 8 connection(s)
I20260927 14:55:40.019338 20967 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6936m5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790520684602251-392-0/minicluster-data/master-0/data/info.pb
I20260927 14:55:40.021968 21035 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 14:55:40.025300 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6936m5/build/debug/bin/kudu as pid 20967
I20260927 14:55:40.025429 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6936m5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790520684602251-392-0/minicluster-data/master-0/wal/instance
I20260927 14:55:40.025687 392 external_mini_cluster.cc:1428] Setting key 07df55e515b8d2474463ba1b7cdeb37e
I20260927 14:55:40.027798 21035 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8241dc76c53648ea8f91ab03993f7045: Bootstrap starting.
I20260927 14:55:40.030294 21035 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8241dc76c53648ea8f91ab03993f7045: Neither blocks nor log segments found. Creating new log.
I20260927 14:55:40.031140 21035 log.cc:824] T 00000000000000000000000000000000 P 8241dc76c53648ea8f91ab03993f7045: Log is configured to *not* fsync() on all Append() calls
I20260927 14:55:40.033357 21035 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8241dc76c53648ea8f91ab03993f7045: No bootstrap required, opened a new log
Sep 27 14:55:40 dist-test-slave-xk1g krb5kdc[20954](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790520939, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260927 14:55:40.036957 21035 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8241dc76c53648ea8f91ab03993f7045 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8241dc76c53648ea8f91ab03993f7045" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37547 } }
I20260927 14:55:40.037249 21035 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8241dc76c53648ea8f91ab03993f7045 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 14:55:40.037329 21035 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8241dc76c53648ea8f91ab03993f7045 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8241dc76c53648ea8f91ab03993f7045, State: Initialized, Role: FOLLOWER
I20260927 14:55:40.037770 21035 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8241dc76c53648ea8f91ab03993f7045 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8241dc76c53648ea8f91ab03993f7045" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37547 } }
I20260927 14:55:40.037928 21035 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8241dc76c53648ea8f91ab03993f7045 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 14:55:40.038015 21035 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8241dc76c53648ea8f91ab03993f7045 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 14:55:40.038117 21035 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8241dc76c53648ea8f91ab03993f7045 [term 0 FOLLOWER]: Advancing to term 1
I20260927 14:55:40.038988 21035 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8241dc76c53648ea8f91ab03993f7045 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8241dc76c53648ea8f91ab03993f7045" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37547 } }
I20260927 14:55:40.039366 21035 leader_election.cc:302] T 00000000000000000000000000000000 P 8241dc76c53648ea8f91ab03993f7045 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8241dc76c53648ea8f91ab03993f7045; no voters:
I20260927 14:55:40.039716 21035 leader_election.cc:288] T 00000000000000000000000000000000 P 8241dc76c53648ea8f91ab03993f7045 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 14:55:40.039809 21040 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8241dc76c53648ea8f91ab03993f7045 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 14:55:40.039995 21040 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8241dc76c53648ea8f91ab03993f7045 [term 1 LEADER]: Becoming Leader. State: Replica: 8241dc76c53648ea8f91ab03993f7045, State: Running, Role: LEADER
I20260927 14:55:40.040262 21038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:55:40.026919 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38669 (local address 127.0.0.1:37547)
0927 14:55:40.027309 (+ 390us) server_negotiation.cc:206] Beginning negotiation
0927 14:55:40.027336 (+ 27us) server_negotiation.cc:399] Waiting for connection header
0927 14:55:40.027371 (+ 35us) server_negotiation.cc:407] Connection header received
0927 14:55:40.028022 (+ 651us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:55:40.028040 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:55:40.028379 (+ 339us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:55:40.028651 (+ 272us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:55:40.029575 (+ 924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:55:40.030640 (+ 1065us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:55:40.031760 (+ 1120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:55:40.032295 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:55:40.032414 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:55:40.035199 (+ 2785us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:55:40.035205 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:55:40.035217 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 14:55:40.035243 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:55:40.036980 (+ 1737us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:55:40.037547 (+ 567us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:55:40.037552 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:55:40.037557 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:55:40.037726 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:55:40.038024 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:55:40.038031 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:55:40.038033 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:55:40.038510 (+ 477us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:55:40.038734 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0927 14:55:40.038982 (+ 248us) server_negotiation.cc:299] Negotiation successful
0927 14:55:40.039309 (+ 327us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":259,"thread_start_us":129,"threads_started":1}
I20260927 14:55:40.040344 21040 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8241dc76c53648ea8f91ab03993f7045 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8241dc76c53648ea8f91ab03993f7045" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37547 } }
I20260927 14:55:40.041005 21035 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8241dc76c53648ea8f91ab03993f7045 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 14:55:40.042079 21041 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8241dc76c53648ea8f91ab03993f7045 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8241dc76c53648ea8f91ab03993f7045" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8241dc76c53648ea8f91ab03993f7045" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37547 } } }
I20260927 14:55:40.042287 21041 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8241dc76c53648ea8f91ab03993f7045 [sys.catalog]: This master's current role is: LEADER
I20260927 14:55:40.043450 21042 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8241dc76c53648ea8f91ab03993f7045 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8241dc76c53648ea8f91ab03993f7045. Latest consensus state: current_term: 1 leader_uuid: "8241dc76c53648ea8f91ab03993f7045" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8241dc76c53648ea8f91ab03993f7045" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37547 } } }
I20260927 14:55:40.043833 21042 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8241dc76c53648ea8f91ab03993f7045 [sys.catalog]: This master's current role is: LEADER
I20260927 14:55:40.043845 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
W20260927 14:55:40.047226 21055 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 8241dc76c53648ea8f91ab03993f7045: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 14:55:40.047320 21055 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 14:55:40.047422 21049 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 14:55:40.048110 21049 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 14:55:40.053259 21049 catalog_manager.cc:1347] Generated new cluster ID: bf012974b35b4b5fb84a08f1f758cacf
I20260927 14:55:40.053324 21049 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 14:55:40.059259 21038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:55:40.045861 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39877 (local address 127.0.0.1:37547)
0927 14:55:40.045981 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0927 14:55:40.045985 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 14:55:40.046217 (+ 232us) server_negotiation.cc:407] Connection header received
0927 14:55:40.046389 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:55:40.046392 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:55:40.046439 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:55:40.046490 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:55:40.047394 (+ 904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:55:40.048173 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:55:40.049392 (+ 1219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:55:40.049738 (+ 346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:55:40.049820 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:55:40.051626 (+ 1806us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:55:40.051630 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:55:40.051633 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 14:55:40.051658 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:55:40.057642 (+ 5984us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:55:40.058077 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:55:40.058081 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:55:40.058083 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:55:40.058164 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:55:40.058442 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:55:40.058445 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:55:40.058447 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:55:40.058614 (+ 167us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:55:40.058748 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0927 14:55:40.058976 (+ 228us) server_negotiation.cc:299] Negotiation successful
0927 14:55:40.059091 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260927 14:55:40.071998 21049 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 14:55:40.073359 21049 catalog_manager.cc:1504] Loading token signing keys...
I20260927 14:55:40.078266 21049 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8241dc76c53648ea8f91ab03993f7045: Generated new TSK 0
I20260927 14:55:40.079028 21049 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 14:55:40.090113 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6936m5/build/debug/bin/kudu with pid 20967
2026-09-27T14:55:40Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (1432 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-task6936m5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790520684602251-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 14:55:40 dist-test-slave-xk1g krb5kdc[21066](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 14:55:40 dist-test-slave-xk1g krb5kdc[21066](info): set up 2 sockets
Sep 27 14:55:40 dist-test-slave-xk1g krb5kdc[21066](info): commencing operation
krb5kdc: starting...
W20260927 14:55:41.121268 392 mini_kdc.cc:121] Time spent starting KDC: real 1.011s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6936m5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790520684602251-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6936m5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790520684602251-392-0/krb5kdc/test-user.keytab.
Sep 27 14:55:41 dist-test-slave-xk1g krb5kdc[21066](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790520941, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T14:55:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T14:55:41Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6936m5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790520684602251-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6936m5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790520684602251-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 14:55:41.239600 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6936m5/build/debug/bin/kudu
/tmp/dist-test-task6936m5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790520684602251-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790520684602251-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790520684602251-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790520684602251-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45753
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790520684602251-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35117
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45753
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790520684602251-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790520684602251-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790520684602251-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 14:55:41.345609 21079 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 14:55:41.345899 21079 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 14:55:41.345979 21079 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 14:55:41.349923 21079 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 14:55:41.350010 21079 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 14:55:41.350047 21079 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 14:55:41.350108 21079 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 14:55:41.350147 21079 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 14:55:41.355306 21079 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35117
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790520684602251-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790520684602251-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45753
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790520684602251-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45753
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790520684602251-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21079
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790520684602251-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:15:17 UTC on ccycloud.quasar-ledxfe.root.comops.site
build id 4676134
I20260927 14:55:41.356317 21079 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 14:55:41.357318 21079 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 14:55:41.365473 21087 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 14:55:41.365561 21084 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 14:55:41.365495 21085 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 14:55:41.365787 21079 server_base.cc:1034] running on GCE node
I20260927 14:55:41.366360 21079 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 14:55:41.367408 21079 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 14:55:41.368594 21079 hybrid_clock.cc:648] HybridClock initialized: now 1790520941368576 us; error 35 us; skew 500 ppm
Sep 27 14:55:41 dist-test-slave-xk1g krb5kdc[21066](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790520941, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 14:55:41.371244 21079 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 14:55:41.372092 21079 webserver.cc:460] Webserver started at http://127.0.0.1:41801/ using document root <none> and password file <none>
I20260927 14:55:41.372634 21079 fs_manager.cc:359] Metadata directory not provided
I20260927 14:55:41.372727 21079 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 14:55:41.372949 21079 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 14:55:41.374917 21079 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6936m5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790520684602251-392-0/minicluster-data/master-0/data/instance:
uuid: "0b93dcd6e547410f81ef6ab5d8f5177c"
format_stamp: "Formatted at 2026-09-27 14:55:41 on dist-test-slave-xk1g"
server_key: "6794772cb1d9020c3e9e07538bad1ba3"
server_key_iv: "647a9d4e768eae48170fe0dbd497ce1c"
server_key_version: "encryptionkey@0"
I20260927 14:55:41.375464 21079 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6936m5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790520684602251-392-0/minicluster-data/master-0/wal/instance:
uuid: "0b93dcd6e547410f81ef6ab5d8f5177c"
format_stamp: "Formatted at 2026-09-27 14:55:41 on dist-test-slave-xk1g"
server_key: "6794772cb1d9020c3e9e07538bad1ba3"
server_key_iv: "647a9d4e768eae48170fe0dbd497ce1c"
server_key_version: "encryptionkey@0"
I20260927 14:55:41.378957 21079 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.001s
I20260927 14:55:41.381253 21094 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 14:55:41.382392 21079 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 14:55:41.382529 21079 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790520684602251-392-0/minicluster-data/master-0/data,/tmp/dist-test-task6936m5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790520684602251-392-0/minicluster-data/master-0/wal
uuid: "0b93dcd6e547410f81ef6ab5d8f5177c"
format_stamp: "Formatted at 2026-09-27 14:55:41 on dist-test-slave-xk1g"
server_key: "6794772cb1d9020c3e9e07538bad1ba3"
server_key_iv: "647a9d4e768eae48170fe0dbd497ce1c"
server_key_version: "encryptionkey@0"
I20260927 14:55:41.382635 21079 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790520684602251-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790520684602251-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790520684602251-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 14:55:41.395326 21079 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 14:55:41.397992 21079 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 14:55:41.398201 21079 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 14:55:41.404968 21079 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45753
I20260927 14:55:41.404994 21146 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45753 every 8 connection(s)
I20260927 14:55:41.405932 21079 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6936m5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790520684602251-392-0/minicluster-data/master-0/data/info.pb
I20260927 14:55:41.408591 21147 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 14:55:41.414111 21147 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0b93dcd6e547410f81ef6ab5d8f5177c: Bootstrap starting.
I20260927 14:55:41.415550 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6936m5/build/debug/bin/kudu as pid 21079
I20260927 14:55:41.415701 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6936m5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790520684602251-392-0/minicluster-data/master-0/wal/instance
I20260927 14:55:41.415974 392 external_mini_cluster.cc:1428] Setting key 4dbe5d069bf3282614b42d79a1873189
I20260927 14:55:41.417018 21147 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0b93dcd6e547410f81ef6ab5d8f5177c: Neither blocks nor log segments found. Creating new log.
I20260927 14:55:41.417897 21147 log.cc:824] T 00000000000000000000000000000000 P 0b93dcd6e547410f81ef6ab5d8f5177c: Log is configured to *not* fsync() on all Append() calls
I20260927 14:55:41.420029 21147 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0b93dcd6e547410f81ef6ab5d8f5177c: No bootstrap required, opened a new log
I20260927 14:55:41.424765 21147 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0b93dcd6e547410f81ef6ab5d8f5177c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b93dcd6e547410f81ef6ab5d8f5177c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45753 } }
I20260927 14:55:41.425117 21147 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0b93dcd6e547410f81ef6ab5d8f5177c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 14:55:41.425184 21147 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0b93dcd6e547410f81ef6ab5d8f5177c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b93dcd6e547410f81ef6ab5d8f5177c, State: Initialized, Role: FOLLOWER
I20260927 14:55:41.425616 21147 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0b93dcd6e547410f81ef6ab5d8f5177c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b93dcd6e547410f81ef6ab5d8f5177c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45753 } }
I20260927 14:55:41.425768 21147 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0b93dcd6e547410f81ef6ab5d8f5177c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 14:55:41.425839 21147 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0b93dcd6e547410f81ef6ab5d8f5177c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 14:55:41.425884 21150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:55:41.417168 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40677 (local address 127.0.0.1:45753)
0927 14:55:41.417697 (+ 529us) server_negotiation.cc:206] Beginning negotiation
0927 14:55:41.417717 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0927 14:55:41.417777 (+ 60us) server_negotiation.cc:407] Connection header received
0927 14:55:41.418759 (+ 982us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:55:41.418787 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:55:41.419321 (+ 534us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:55:41.419747 (+ 426us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:55:41.420173 (+ 426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:55:41.421288 (+ 1115us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:55:41.422548 (+ 1260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:55:41.423032 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:55:41.423160 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:55:41.424067 (+ 907us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:55:41.424071 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:55:41.424084 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 14:55:41.424492 (+ 408us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:55:41.424547 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 14:55:41.424569 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:55:41.424642 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0927 14:55:41.424811 (+ 169us) server_negotiation.cc:299] Negotiation successful
0927 14:55:41.425036 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":330,"thread_start_us":139,"threads_started":1}
I20260927 14:55:41.425951 21147 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0b93dcd6e547410f81ef6ab5d8f5177c [term 0 FOLLOWER]: Advancing to term 1
I20260927 14:55:41.426910 21147 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0b93dcd6e547410f81ef6ab5d8f5177c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b93dcd6e547410f81ef6ab5d8f5177c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45753 } }
I20260927 14:55:41.427260 21147 leader_election.cc:302] T 00000000000000000000000000000000 P 0b93dcd6e547410f81ef6ab5d8f5177c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0b93dcd6e547410f81ef6ab5d8f5177c; no voters:
I20260927 14:55:41.427621 21147 leader_election.cc:288] T 00000000000000000000000000000000 P 0b93dcd6e547410f81ef6ab5d8f5177c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 14:55:41.427690 21152 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0b93dcd6e547410f81ef6ab5d8f5177c [term 1 FOLLOWER]: Leader election won for term 1
I20260927 14:55:41.427892 21152 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0b93dcd6e547410f81ef6ab5d8f5177c [term 1 LEADER]: Becoming Leader. State: Replica: 0b93dcd6e547410f81ef6ab5d8f5177c, State: Running, Role: LEADER
I20260927 14:55:41.428231 21152 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0b93dcd6e547410f81ef6ab5d8f5177c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b93dcd6e547410f81ef6ab5d8f5177c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45753 } }
I20260927 14:55:41.428649 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 14:55:41.428953 21147 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0b93dcd6e547410f81ef6ab5d8f5177c [sys.catalog]: configured and running, proceeding with master startup.
I20260927 14:55:41.429363 21152 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0b93dcd6e547410f81ef6ab5d8f5177c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0b93dcd6e547410f81ef6ab5d8f5177c. Latest consensus state: current_term: 1 leader_uuid: "0b93dcd6e547410f81ef6ab5d8f5177c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b93dcd6e547410f81ef6ab5d8f5177c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45753 } } }
I20260927 14:55:41.429478 21152 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0b93dcd6e547410f81ef6ab5d8f5177c [sys.catalog]: This master's current role is: LEADER
I20260927 14:55:41.429808 21154 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0b93dcd6e547410f81ef6ab5d8f5177c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0b93dcd6e547410f81ef6ab5d8f5177c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b93dcd6e547410f81ef6ab5d8f5177c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45753 } } }
I20260927 14:55:41.429890 21154 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0b93dcd6e547410f81ef6ab5d8f5177c [sys.catalog]: This master's current role is: LEADER
W20260927 14:55:41.434331 21172 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 0b93dcd6e547410f81ef6ab5d8f5177c: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 14:55:41.434422 21172 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 14:55:41.434491 21156 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 14:55:41.435148 21150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:55:41.431172 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41169 (local address 127.0.0.1:45753)
0927 14:55:41.431312 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0927 14:55:41.431317 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 14:55:41.431330 (+ 13us) server_negotiation.cc:407] Connection header received
0927 14:55:41.431363 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:55:41.431365 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:55:41.431403 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:55:41.431476 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:55:41.431874 (+ 398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:55:41.432404 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:55:41.433415 (+ 1011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:55:41.433733 (+ 318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:55:41.433845 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:55:41.434727 (+ 882us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:55:41.434729 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:55:41.434732 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 14:55:41.434742 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:55:41.434765 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 14:55:41.434775 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:55:41.434830 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0927 14:55:41.434957 (+ 127us) server_negotiation.cc:299] Negotiation successful
0927 14:55:41.435002 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260927 14:55:41.435448 21156 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 14:55:41.441283 21156 catalog_manager.cc:1347] Generated new cluster ID: 78ea4c6d3309406889cd55ff47793cad
I20260927 14:55:41.441367 21156 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 14:55:41.473918 21156 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 14:55:41.474983 21156 catalog_manager.cc:1504] Loading token signing keys...
I20260927 14:55:41.484234 21156 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0b93dcd6e547410f81ef6ab5d8f5177c: Generated new TSK 0
I20260927 14:55:41.484946 21156 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 14:55:41.491056 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6936m5/build/debug/bin/kudu with pid 21079
2026-09-27T14:55:41Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1400 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-task6936m5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790520684602251-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 14:55:41 dist-test-slave-xk1g krb5kdc[21178](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 14:55:41 dist-test-slave-xk1g krb5kdc[21178](info): set up 2 sockets
Sep 27 14:55:41 dist-test-slave-xk1g krb5kdc[21178](info): commencing operation
krb5kdc: starting...
W20260927 14:55:42.541664 392 mini_kdc.cc:121] Time spent starting KDC: real 1.031s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6936m5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790520684602251-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6936m5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790520684602251-392-0/krb5kdc/test-user.keytab.
Sep 27 14:55:42 dist-test-slave-xk1g krb5kdc[21178](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790520942, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T14:55:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T14:55:42Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6936m5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790520684602251-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6936m5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790520684602251-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 14:55:42.660393 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6936m5/build/debug/bin/kudu
/tmp/dist-test-task6936m5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790520684602251-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790520684602251-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790520684602251-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790520684602251-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38597
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790520684602251-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39553
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38597
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790520684602251-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790520684602251-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790520684602251-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 14:55:42.768505 21191 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 14:55:42.768811 21191 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 14:55:42.768896 21191 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 14:55:42.772921 21191 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 14:55:42.773020 21191 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 14:55:42.773054 21191 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 14:55:42.773098 21191 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 14:55:42.773152 21191 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 14:55:42.778432 21191 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39553
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790520684602251-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790520684602251-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38597
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790520684602251-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38597
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790520684602251-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21191
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790520684602251-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:15:17 UTC on ccycloud.quasar-ledxfe.root.comops.site
build id 4676134
I20260927 14:55:42.779500 21191 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 14:55:42.780546 21191 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 14:55:42.789036 21196 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 14:55:42.789111 21197 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 14:55:42.788990 21199 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 14:55:42.789323 21191 server_base.cc:1034] running on GCE node
I20260927 14:55:42.789914 21191 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 14:55:42.790977 21191 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 14:55:42.792183 21191 hybrid_clock.cc:648] HybridClock initialized: now 1790520942792175 us; error 29 us; skew 500 ppm
Sep 27 14:55:42 dist-test-slave-xk1g krb5kdc[21178](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790520942, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 14:55:42.794989 21191 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 14:55:42.795863 21191 webserver.cc:460] Webserver started at http://127.0.0.1:38719/ using document root <none> and password file <none>
I20260927 14:55:42.796429 21191 fs_manager.cc:359] Metadata directory not provided
I20260927 14:55:42.796518 21191 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 14:55:42.796746 21191 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 14:55:42.798827 21191 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6936m5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790520684602251-392-0/minicluster-data/master-0/data/instance:
uuid: "8c67cec3b9ae4467a09d5c0a9ee02171"
format_stamp: "Formatted at 2026-09-27 14:55:42 on dist-test-slave-xk1g"
server_key: "80d5d8846ebf473d9424f67434433585"
server_key_iv: "44cefc6a42795d0c8d81f1b2281af8ad"
server_key_version: "encryptionkey@0"
I20260927 14:55:42.799389 21191 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6936m5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790520684602251-392-0/minicluster-data/master-0/wal/instance:
uuid: "8c67cec3b9ae4467a09d5c0a9ee02171"
format_stamp: "Formatted at 2026-09-27 14:55:42 on dist-test-slave-xk1g"
server_key: "80d5d8846ebf473d9424f67434433585"
server_key_iv: "44cefc6a42795d0c8d81f1b2281af8ad"
server_key_version: "encryptionkey@0"
I20260927 14:55:42.803062 21191 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260927 14:55:42.805418 21206 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 14:55:42.806644 21191 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 14:55:42.806767 21191 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790520684602251-392-0/minicluster-data/master-0/data,/tmp/dist-test-task6936m5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790520684602251-392-0/minicluster-data/master-0/wal
uuid: "8c67cec3b9ae4467a09d5c0a9ee02171"
format_stamp: "Formatted at 2026-09-27 14:55:42 on dist-test-slave-xk1g"
server_key: "80d5d8846ebf473d9424f67434433585"
server_key_iv: "44cefc6a42795d0c8d81f1b2281af8ad"
server_key_version: "encryptionkey@0"
I20260927 14:55:42.806852 21191 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790520684602251-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790520684602251-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790520684602251-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 14:55:42.811277 21191 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 14:55:42.814023 21191 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 14:55:42.814158 21191 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 14:55:42.820858 21191 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38597
I20260927 14:55:42.820875 21258 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38597 every 8 connection(s)
I20260927 14:55:42.821784 21191 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6936m5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790520684602251-392-0/minicluster-data/master-0/data/info.pb
I20260927 14:55:42.824448 21259 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 14:55:42.826154 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6936m5/build/debug/bin/kudu as pid 21191
I20260927 14:55:42.826303 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6936m5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790520684602251-392-0/minicluster-data/master-0/wal/instance
I20260927 14:55:42.826583 392 external_mini_cluster.cc:1428] Setting key aafff2ae44956d17be0edc5e1e691faf
I20260927 14:55:42.831229 21259 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8c67cec3b9ae4467a09d5c0a9ee02171: Bootstrap starting.
I20260927 14:55:42.832197 21262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:55:42.827973 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47913 (local address 127.0.0.1:38597)
0927 14:55:42.828389 (+ 416us) server_negotiation.cc:206] Beginning negotiation
0927 14:55:42.828407 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0927 14:55:42.828450 (+ 43us) server_negotiation.cc:407] Connection header received
0927 14:55:42.829343 (+ 893us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:55:42.829379 (+ 36us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:55:42.829738 (+ 359us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:55:42.830112 (+ 374us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:55:42.830790 (+ 678us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:55:42.830795 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:55:42.830812 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 14:55:42.830825 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:55:42.830888 (+ 63us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 14:55:42.830914 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:55:42.831049 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0927 14:55:42.831173 (+ 124us) server_negotiation.cc:299] Negotiation successful
0927 14:55:42.831321 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":266,"thread_start_us":116,"threads_started":1}
I20260927 14:55:42.833828 21259 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8c67cec3b9ae4467a09d5c0a9ee02171: Neither blocks nor log segments found. Creating new log.
I20260927 14:55:42.834717 21259 log.cc:824] T 00000000000000000000000000000000 P 8c67cec3b9ae4467a09d5c0a9ee02171: Log is configured to *not* fsync() on all Append() calls
I20260927 14:55:42.835152 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 14:55:42.836735 21259 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8c67cec3b9ae4467a09d5c0a9ee02171: No bootstrap required, opened a new log
I20260927 14:55:42.838780 21262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:55:42.837205 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45813 (local address 127.0.0.1:38597)
0927 14:55:42.837325 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0927 14:55:42.837330 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 14:55:42.837513 (+ 183us) server_negotiation.cc:407] Connection header received
0927 14:55:42.837624 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:55:42.837626 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:55:42.837653 (+ 27us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:55:42.837717 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:55:42.838388 (+ 671us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:55:42.838391 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:55:42.838392 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 14:55:42.838402 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:55:42.838424 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 14:55:42.838434 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:55:42.838500 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0927 14:55:42.838594 (+ 94us) server_negotiation.cc:299] Negotiation successful
0927 14:55:42.838624 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260927 14:55:42.841243 21259 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8c67cec3b9ae4467a09d5c0a9ee02171 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c67cec3b9ae4467a09d5c0a9ee02171" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38597 } }
I20260927 14:55:42.841477 21259 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8c67cec3b9ae4467a09d5c0a9ee02171 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 14:55:42.841543 21259 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8c67cec3b9ae4467a09d5c0a9ee02171 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8c67cec3b9ae4467a09d5c0a9ee02171, State: Initialized, Role: FOLLOWER
I20260927 14:55:42.841938 21259 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8c67cec3b9ae4467a09d5c0a9ee02171 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c67cec3b9ae4467a09d5c0a9ee02171" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38597 } }
I20260927 14:55:42.842082 21259 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8c67cec3b9ae4467a09d5c0a9ee02171 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 14:55:42.842164 21259 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8c67cec3b9ae4467a09d5c0a9ee02171 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 14:55:42.842262 21259 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8c67cec3b9ae4467a09d5c0a9ee02171 [term 0 FOLLOWER]: Advancing to term 1
I20260927 14:55:42.843266 21259 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8c67cec3b9ae4467a09d5c0a9ee02171 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c67cec3b9ae4467a09d5c0a9ee02171" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38597 } }
I20260927 14:55:42.843595 21259 leader_election.cc:302] T 00000000000000000000000000000000 P 8c67cec3b9ae4467a09d5c0a9ee02171 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8c67cec3b9ae4467a09d5c0a9ee02171; no voters:
I20260927 14:55:42.843888 21259 leader_election.cc:288] T 00000000000000000000000000000000 P 8c67cec3b9ae4467a09d5c0a9ee02171 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 14:55:42.844012 21269 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8c67cec3b9ae4467a09d5c0a9ee02171 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 14:55:42.844245 21269 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8c67cec3b9ae4467a09d5c0a9ee02171 [term 1 LEADER]: Becoming Leader. State: Replica: 8c67cec3b9ae4467a09d5c0a9ee02171, State: Running, Role: LEADER
I20260927 14:55:42.844568 21269 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8c67cec3b9ae4467a09d5c0a9ee02171 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c67cec3b9ae4467a09d5c0a9ee02171" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38597 } }
I20260927 14:55:42.845155 21259 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8c67cec3b9ae4467a09d5c0a9ee02171 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 14:55:42.846161 21270 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8c67cec3b9ae4467a09d5c0a9ee02171 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8c67cec3b9ae4467a09d5c0a9ee02171" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c67cec3b9ae4467a09d5c0a9ee02171" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38597 } } }
I20260927 14:55:42.846299 21270 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8c67cec3b9ae4467a09d5c0a9ee02171 [sys.catalog]: This master's current role is: LEADER
I20260927 14:55:42.846621 21271 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8c67cec3b9ae4467a09d5c0a9ee02171 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8c67cec3b9ae4467a09d5c0a9ee02171. Latest consensus state: current_term: 1 leader_uuid: "8c67cec3b9ae4467a09d5c0a9ee02171" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c67cec3b9ae4467a09d5c0a9ee02171" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38597 } } }
I20260927 14:55:42.846683 21278 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 14:55:42.846710 21271 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8c67cec3b9ae4467a09d5c0a9ee02171 [sys.catalog]: This master's current role is: LEADER
I20260927 14:55:42.849560 21278 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 14:55:42.855094 21278 catalog_manager.cc:1347] Generated new cluster ID: 6c368de13a7546129c671c3474720dd1
I20260927 14:55:42.855171 21278 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 14:55:42.863904 21278 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 14:55:42.864188 21278 catalog_manager.cc:1504] Loading token signing keys...
I20260927 14:55:42.867731 21278 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8c67cec3b9ae4467a09d5c0a9ee02171: Generated new TSK 0
I20260927 14:55:42.868307 21278 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 14:55:42.880946 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6936m5/build/debug/bin/kudu with pid 21191
2026-09-27T14:55:42Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1390 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-task6936m5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790520684602251-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 14:55:42 dist-test-slave-xk1g krb5kdc[21290](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 14:55:42 dist-test-slave-xk1g krb5kdc[21290](info): set up 2 sockets
Sep 27 14:55:42 dist-test-slave-xk1g krb5kdc[21290](info): commencing operation
krb5kdc: starting...
W20260927 14:55:43.921232 392 mini_kdc.cc:121] Time spent starting KDC: real 1.020s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6936m5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790520684602251-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6936m5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790520684602251-392-0/krb5kdc/test-user.keytab.
Sep 27 14:55:44 dist-test-slave-xk1g krb5kdc[21290](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790520944, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T14:55:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T14:55:44Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6936m5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790520684602251-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6936m5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790520684602251-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 14:55:44.040516 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6936m5/build/debug/bin/kudu
/tmp/dist-test-task6936m5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790520684602251-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790520684602251-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790520684602251-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790520684602251-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46817
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790520684602251-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37549
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:46817
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790520684602251-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790520684602251-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790520684602251-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 14:55:44.148967 21303 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 14:55:44.149282 21303 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 14:55:44.149359 21303 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 14:55:44.153319 21303 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 14:55:44.153414 21303 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 14:55:44.153455 21303 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 14:55:44.153491 21303 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 14:55:44.153527 21303 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 14:55:44.158718 21303 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37549
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790520684602251-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790520684602251-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:46817
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790520684602251-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46817
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790520684602251-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21303
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790520684602251-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:15:17 UTC on ccycloud.quasar-ledxfe.root.comops.site
build id 4676134
I20260927 14:55:44.159790 21303 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 14:55:44.160797 21303 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 14:55:44.169014 21309 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 14:55:44.169034 21311 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 14:55:44.169021 21308 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 14:55:44.169263 21303 server_base.cc:1034] running on GCE node
I20260927 14:55:44.169900 21303 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 14:55:44.170949 21303 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 14:55:44.172140 21303 hybrid_clock.cc:648] HybridClock initialized: now 1790520944172115 us; error 38 us; skew 500 ppm
Sep 27 14:55:44 dist-test-slave-xk1g krb5kdc[21290](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790520944, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 14:55:44.174840 21303 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 14:55:44.175724 21303 webserver.cc:460] Webserver started at http://127.0.0.1:42587/ using document root <none> and password file <none>
I20260927 14:55:44.176261 21303 fs_manager.cc:359] Metadata directory not provided
I20260927 14:55:44.176349 21303 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 14:55:44.176579 21303 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 14:55:44.178612 21303 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6936m5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790520684602251-392-0/minicluster-data/master-0/data/instance:
uuid: "c2b88ba214344212a257fa223f422b94"
format_stamp: "Formatted at 2026-09-27 14:55:44 on dist-test-slave-xk1g"
server_key: "ab2a9ab7a7cace330d4ab488122ab026"
server_key_iv: "4e725866acf41544343166363e5126ea"
server_key_version: "encryptionkey@0"
I20260927 14:55:44.179159 21303 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6936m5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790520684602251-392-0/minicluster-data/master-0/wal/instance:
uuid: "c2b88ba214344212a257fa223f422b94"
format_stamp: "Formatted at 2026-09-27 14:55:44 on dist-test-slave-xk1g"
server_key: "ab2a9ab7a7cace330d4ab488122ab026"
server_key_iv: "4e725866acf41544343166363e5126ea"
server_key_version: "encryptionkey@0"
I20260927 14:55:44.182721 21303 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260927 14:55:44.185102 21318 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 14:55:44.186399 21303 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 14:55:44.186554 21303 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790520684602251-392-0/minicluster-data/master-0/data,/tmp/dist-test-task6936m5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790520684602251-392-0/minicluster-data/master-0/wal
uuid: "c2b88ba214344212a257fa223f422b94"
format_stamp: "Formatted at 2026-09-27 14:55:44 on dist-test-slave-xk1g"
server_key: "ab2a9ab7a7cace330d4ab488122ab026"
server_key_iv: "4e725866acf41544343166363e5126ea"
server_key_version: "encryptionkey@0"
I20260927 14:55:44.186671 21303 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790520684602251-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790520684602251-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790520684602251-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 14:55:44.198472 21303 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 14:55:44.201453 21303 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 14:55:44.201647 21303 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 14:55:44.208673 21303 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:46817
I20260927 14:55:44.208688 21370 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:46817 every 8 connection(s)
I20260927 14:55:44.209643 21303 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6936m5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790520684602251-392-0/minicluster-data/master-0/data/info.pb
I20260927 14:55:44.212239 21371 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 14:55:44.216570 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6936m5/build/debug/bin/kudu as pid 21303
I20260927 14:55:44.216715 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6936m5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790520684602251-392-0/minicluster-data/master-0/wal/instance
I20260927 14:55:44.216989 392 external_mini_cluster.cc:1428] Setting key 8100b09d8de0e41927609ea238009a0c
I20260927 14:55:44.217638 21371 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c2b88ba214344212a257fa223f422b94: Bootstrap starting.
I20260927 14:55:44.219905 21371 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c2b88ba214344212a257fa223f422b94: Neither blocks nor log segments found. Creating new log.
I20260927 14:55:44.220602 21371 log.cc:824] T 00000000000000000000000000000000 P c2b88ba214344212a257fa223f422b94: Log is configured to *not* fsync() on all Append() calls
I20260927 14:55:44.222698 21371 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c2b88ba214344212a257fa223f422b94: No bootstrap required, opened a new log
Sep 27 14:55:44 dist-test-slave-xk1g krb5kdc[21290](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790520944, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260927 14:55:44.226789 21371 raft_consensus.cc:348] T 00000000000000000000000000000000 P c2b88ba214344212a257fa223f422b94 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c2b88ba214344212a257fa223f422b94" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46817 } }
I20260927 14:55:44.227041 21371 raft_consensus.cc:374] T 00000000000000000000000000000000 P c2b88ba214344212a257fa223f422b94 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 14:55:44.227116 21371 raft_consensus.cc:729] T 00000000000000000000000000000000 P c2b88ba214344212a257fa223f422b94 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c2b88ba214344212a257fa223f422b94, State: Initialized, Role: FOLLOWER
I20260927 14:55:44.227525 21371 consensus_queue.cc:260] T 00000000000000000000000000000000 P c2b88ba214344212a257fa223f422b94 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c2b88ba214344212a257fa223f422b94" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46817 } }
I20260927 14:55:44.227679 21371 raft_consensus.cc:388] T 00000000000000000000000000000000 P c2b88ba214344212a257fa223f422b94 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 14:55:44.227762 21371 raft_consensus.cc:482] T 00000000000000000000000000000000 P c2b88ba214344212a257fa223f422b94 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 14:55:44.227864 21371 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c2b88ba214344212a257fa223f422b94 [term 0 FOLLOWER]: Advancing to term 1
I20260927 14:55:44.228683 21371 raft_consensus.cc:504] T 00000000000000000000000000000000 P c2b88ba214344212a257fa223f422b94 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c2b88ba214344212a257fa223f422b94" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46817 } }
I20260927 14:55:44.229007 21371 leader_election.cc:302] T 00000000000000000000000000000000 P c2b88ba214344212a257fa223f422b94 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c2b88ba214344212a257fa223f422b94; no voters:
I20260927 14:55:44.229348 21371 leader_election.cc:288] T 00000000000000000000000000000000 P c2b88ba214344212a257fa223f422b94 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 14:55:44.229440 21376 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c2b88ba214344212a257fa223f422b94 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 14:55:44.229599 21376 raft_consensus.cc:686] T 00000000000000000000000000000000 P c2b88ba214344212a257fa223f422b94 [term 1 LEADER]: Becoming Leader. State: Replica: c2b88ba214344212a257fa223f422b94, State: Running, Role: LEADER
I20260927 14:55:44.229882 21376 consensus_queue.cc:237] T 00000000000000000000000000000000 P c2b88ba214344212a257fa223f422b94 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c2b88ba214344212a257fa223f422b94" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46817 } }
I20260927 14:55:44.230557 21371 sys_catalog.cc:564] T 00000000000000000000000000000000 P c2b88ba214344212a257fa223f422b94 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 14:55:44.231588 21377 sys_catalog.cc:455] T 00000000000000000000000000000000 P c2b88ba214344212a257fa223f422b94 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c2b88ba214344212a257fa223f422b94" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c2b88ba214344212a257fa223f422b94" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46817 } } }
I20260927 14:55:44.231616 21378 sys_catalog.cc:455] T 00000000000000000000000000000000 P c2b88ba214344212a257fa223f422b94 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c2b88ba214344212a257fa223f422b94. Latest consensus state: current_term: 1 leader_uuid: "c2b88ba214344212a257fa223f422b94" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c2b88ba214344212a257fa223f422b94" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46817 } } }
I20260927 14:55:44.231729 21377 sys_catalog.cc:458] T 00000000000000000000000000000000 P c2b88ba214344212a257fa223f422b94 [sys.catalog]: This master's current role is: LEADER
I20260927 14:55:44.231741 21378 sys_catalog.cc:458] T 00000000000000000000000000000000 P c2b88ba214344212a257fa223f422b94 [sys.catalog]: This master's current role is: LEADER
I20260927 14:55:44.232213 21374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:55:44.218317 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.55:49777 (local address 127.0.0.1:46817)
0927 14:55:44.218712 (+ 395us) server_negotiation.cc:206] Beginning negotiation
0927 14:55:44.218726 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 14:55:44.218761 (+ 35us) server_negotiation.cc:407] Connection header received
0927 14:55:44.219361 (+ 600us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:55:44.219376 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:55:44.219702 (+ 326us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:55:44.220041 (+ 339us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:55:44.220854 (+ 813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:55:44.222282 (+ 1428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:55:44.223153 (+ 871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:55:44.223807 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:55:44.223955 (+ 148us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:55:44.226603 (+ 2648us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:55:44.226609 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:55:44.226624 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 14:55:44.226653 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:55:44.228583 (+ 1930us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:55:44.229140 (+ 557us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:55:44.229147 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:55:44.229156 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:55:44.229275 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:55:44.229622 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:55:44.229626 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:55:44.229629 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:55:44.230212 (+ 583us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:55:44.230429 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0927 14:55:44.230775 (+ 346us) server_negotiation.cc:299] Negotiation successful
0927 14:55:44.231014 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":266,"thread_start_us":110,"threads_started":1}
W20260927 14:55:44.235311 21391 catalog_manager.cc:1558] T 00000000000000000000000000000000 P c2b88ba214344212a257fa223f422b94: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 14:55:44.235392 21391 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 14:55:44.235489 21385 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 14:55:44.235481 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 14:55:44.236191 21385 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 14:55:44.242336 21385 catalog_manager.cc:1347] Generated new cluster ID: ce8afb0f078345ca9f482ccdb8659687
I20260927 14:55:44.242462 21385 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 14:55:44.245828 21374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:55:44.237708 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.55:37609 (local address 127.0.0.1:46817)
0927 14:55:44.237827 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0927 14:55:44.237831 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 14:55:44.238132 (+ 301us) server_negotiation.cc:407] Connection header received
0927 14:55:44.238289 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:55:44.238292 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:55:44.238342 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:55:44.238421 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:55:44.239036 (+ 615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:55:44.239557 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:55:44.240365 (+ 808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:55:44.240608 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:55:44.240761 (+ 153us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:55:44.242160 (+ 1399us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:55:44.242163 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:55:44.242166 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 14:55:44.242190 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:55:44.243928 (+ 1738us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:55:44.244486 (+ 558us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:55:44.244489 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:55:44.244492 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:55:44.244568 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:55:44.244930 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:55:44.244934 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:55:44.244936 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:55:44.245137 (+ 201us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:55:44.245255 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0927 14:55:44.245573 (+ 318us) server_negotiation.cc:299] Negotiation successful
0927 14:55:44.245656 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":45}
I20260927 14:55:44.249858 21385 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 14:55:44.250764 21385 catalog_manager.cc:1504] Loading token signing keys...
I20260927 14:55:44.261929 21385 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c2b88ba214344212a257fa223f422b94: Generated new TSK 0
I20260927 14:55:44.262681 21385 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 14:55:44.271106 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6936m5/build/debug/bin/kudu with pid 21303
W20260927 14:55:44.276410 21288 connection.cc:535] client connection to 127.0.0.1:46817 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:46817): Connection reset by peer (error 104)
2026-09-27T14:55:44Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (1390 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-task6936m5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790520684602251-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 14:55:44 dist-test-slave-xk1g krb5kdc[21402](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 14:55:44 dist-test-slave-xk1g krb5kdc[21402](info): set up 2 sockets
Sep 27 14:55:44 dist-test-slave-xk1g krb5kdc[21402](info): commencing operation
krb5kdc: starting...
W20260927 14:55:45.313324 392 mini_kdc.cc:121] Time spent starting KDC: real 1.023s user 0.004s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6936m5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790520684602251-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6936m5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790520684602251-392-0/krb5kdc/test-user.keytab.
Sep 27 14:55:45 dist-test-slave-xk1g krb5kdc[21402](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790520945, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T14:55:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T14:55:45Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6936m5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790520684602251-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6936m5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790520684602251-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 14:55:45.432336 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6936m5/build/debug/bin/kudu
/tmp/dist-test-task6936m5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790520684602251-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790520684602251-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790520684602251-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790520684602251-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46217
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790520684602251-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43061
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:46217
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790520684602251-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790520684602251-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790520684602251-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 14:55:45.537685 21415 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 14:55:45.537938 21415 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 14:55:45.537977 21415 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 14:55:45.541872 21415 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 14:55:45.541930 21415 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 14:55:45.541949 21415 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 14:55:45.541965 21415 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 14:55:45.541980 21415 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 14:55:45.547260 21415 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43061
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790520684602251-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790520684602251-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:46217
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790520684602251-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46217
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790520684602251-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21415
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790520684602251-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:15:17 UTC on ccycloud.quasar-ledxfe.root.comops.site
build id 4676134
I20260927 14:55:45.548174 21415 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 14:55:45.549124 21415 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 14:55:45.557245 21423 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 14:55:45.557250 21421 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 14:55:45.557299 21420 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 14:55:45.557565 21415 server_base.cc:1034] running on GCE node
I20260927 14:55:45.558053 21415 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 14:55:45.559108 21415 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 14:55:45.560268 21415 hybrid_clock.cc:648] HybridClock initialized: now 1790520945560247 us; error 33 us; skew 500 ppm
Sep 27 14:55:45 dist-test-slave-xk1g krb5kdc[21402](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790520945, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 14:55:45.567106 21415 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 14:55:45.567948 21415 webserver.cc:460] Webserver started at http://127.0.0.1:35477/ using document root <none> and password file <none>
I20260927 14:55:45.568475 21415 fs_manager.cc:359] Metadata directory not provided
I20260927 14:55:45.568531 21415 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 14:55:45.568756 21415 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 14:55:45.570761 21415 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6936m5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790520684602251-392-0/minicluster-data/master-0/data/instance:
uuid: "353630ae8f054bb8ba359f332561f032"
format_stamp: "Formatted at 2026-09-27 14:55:45 on dist-test-slave-xk1g"
server_key: "5e36efefa770603c4a92b22fe3047929"
server_key_iv: "5262dec09ecbeb36779814bb2c11b374"
server_key_version: "encryptionkey@0"
I20260927 14:55:45.571300 21415 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6936m5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790520684602251-392-0/minicluster-data/master-0/wal/instance:
uuid: "353630ae8f054bb8ba359f332561f032"
format_stamp: "Formatted at 2026-09-27 14:55:45 on dist-test-slave-xk1g"
server_key: "5e36efefa770603c4a92b22fe3047929"
server_key_iv: "5262dec09ecbeb36779814bb2c11b374"
server_key_version: "encryptionkey@0"
I20260927 14:55:45.574856 21415 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260927 14:55:45.577129 21430 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 14:55:45.578303 21415 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 14:55:45.578435 21415 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790520684602251-392-0/minicluster-data/master-0/data,/tmp/dist-test-task6936m5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790520684602251-392-0/minicluster-data/master-0/wal
uuid: "353630ae8f054bb8ba359f332561f032"
format_stamp: "Formatted at 2026-09-27 14:55:45 on dist-test-slave-xk1g"
server_key: "5e36efefa770603c4a92b22fe3047929"
server_key_iv: "5262dec09ecbeb36779814bb2c11b374"
server_key_version: "encryptionkey@0"
I20260927 14:55:45.578569 21415 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790520684602251-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790520684602251-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790520684602251-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 14:55:45.600117 21415 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 14:55:45.603467 21415 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 14:55:45.603672 21415 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 14:55:45.610591 21415 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:46217
I20260927 14:55:45.610616 21482 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:46217 every 8 connection(s)
I20260927 14:55:45.611521 21415 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6936m5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790520684602251-392-0/minicluster-data/master-0/data/info.pb
I20260927 14:55:45.614212 21483 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 14:55:45.618350 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6936m5/build/debug/bin/kudu as pid 21415
I20260927 14:55:45.618486 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6936m5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790520684602251-392-0/minicluster-data/master-0/wal/instance
I20260927 14:55:45.618736 392 external_mini_cluster.cc:1428] Setting key 741cc5c58d5a4a1660b89805c92e5303
I20260927 14:55:45.619709 21483 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 353630ae8f054bb8ba359f332561f032: Bootstrap starting.
I20260927 14:55:45.622324 21483 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 353630ae8f054bb8ba359f332561f032: Neither blocks nor log segments found. Creating new log.
I20260927 14:55:45.623185 21483 log.cc:824] T 00000000000000000000000000000000 P 353630ae8f054bb8ba359f332561f032: Log is configured to *not* fsync() on all Append() calls
I20260927 14:55:45.625109 21483 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 353630ae8f054bb8ba359f332561f032: No bootstrap required, opened a new log
I20260927 14:55:45.628558 21486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:55:45.620121 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.55:36807 (local address 127.0.0.1:46217)
0927 14:55:45.620539 (+ 418us) server_negotiation.cc:206] Beginning negotiation
0927 14:55:45.620557 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0927 14:55:45.620599 (+ 42us) server_negotiation.cc:407] Connection header received
0927 14:55:45.621457 (+ 858us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:55:45.621480 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:55:45.621888 (+ 408us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:55:45.622242 (+ 354us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:55:45.622690 (+ 448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:55:45.624010 (+ 1320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:55:45.624903 (+ 893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:55:45.625602 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:55:45.625761 (+ 159us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:55:45.626549 (+ 788us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:55:45.626556 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:55:45.626573 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 14:55:45.627053 (+ 480us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:55:45.627114 (+ 61us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 14:55:45.627142 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:55:45.627271 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0927 14:55:45.627444 (+ 173us) server_negotiation.cc:299] Negotiation successful
0927 14:55:45.627571 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":280,"thread_start_us":109,"threads_started":1}
I20260927 14:55:45.629380 21483 raft_consensus.cc:348] T 00000000000000000000000000000000 P 353630ae8f054bb8ba359f332561f032 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "353630ae8f054bb8ba359f332561f032" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46217 } }
I20260927 14:55:45.629644 21483 raft_consensus.cc:374] T 00000000000000000000000000000000 P 353630ae8f054bb8ba359f332561f032 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 14:55:45.629709 21483 raft_consensus.cc:729] T 00000000000000000000000000000000 P 353630ae8f054bb8ba359f332561f032 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 353630ae8f054bb8ba359f332561f032, State: Initialized, Role: FOLLOWER
I20260927 14:55:45.630100 21483 consensus_queue.cc:260] T 00000000000000000000000000000000 P 353630ae8f054bb8ba359f332561f032 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "353630ae8f054bb8ba359f332561f032" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46217 } }
I20260927 14:55:45.630242 21483 raft_consensus.cc:388] T 00000000000000000000000000000000 P 353630ae8f054bb8ba359f332561f032 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 14:55:45.630321 21483 raft_consensus.cc:482] T 00000000000000000000000000000000 P 353630ae8f054bb8ba359f332561f032 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 14:55:45.630419 21483 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 353630ae8f054bb8ba359f332561f032 [term 0 FOLLOWER]: Advancing to term 1
I20260927 14:55:45.631415 21483 raft_consensus.cc:504] T 00000000000000000000000000000000 P 353630ae8f054bb8ba359f332561f032 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "353630ae8f054bb8ba359f332561f032" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46217 } }
I20260927 14:55:45.631512 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 14:55:45.631747 21483 leader_election.cc:302] T 00000000000000000000000000000000 P 353630ae8f054bb8ba359f332561f032 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 353630ae8f054bb8ba359f332561f032; no voters:
I20260927 14:55:45.632025 21483 leader_election.cc:288] T 00000000000000000000000000000000 P 353630ae8f054bb8ba359f332561f032 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 14:55:45.632131 21488 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 353630ae8f054bb8ba359f332561f032 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 14:55:45.632350 21488 raft_consensus.cc:686] T 00000000000000000000000000000000 P 353630ae8f054bb8ba359f332561f032 [term 1 LEADER]: Becoming Leader. State: Replica: 353630ae8f054bb8ba359f332561f032, State: Running, Role: LEADER
I20260927 14:55:45.632649 21488 consensus_queue.cc:237] T 00000000000000000000000000000000 P 353630ae8f054bb8ba359f332561f032 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "353630ae8f054bb8ba359f332561f032" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46217 } }
I20260927 14:55:45.633302 21483 sys_catalog.cc:564] T 00000000000000000000000000000000 P 353630ae8f054bb8ba359f332561f032 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 14:55:45.633893 21490 sys_catalog.cc:455] T 00000000000000000000000000000000 P 353630ae8f054bb8ba359f332561f032 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 353630ae8f054bb8ba359f332561f032. Latest consensus state: current_term: 1 leader_uuid: "353630ae8f054bb8ba359f332561f032" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "353630ae8f054bb8ba359f332561f032" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46217 } } }
I20260927 14:55:45.634024 21490 sys_catalog.cc:458] T 00000000000000000000000000000000 P 353630ae8f054bb8ba359f332561f032 [sys.catalog]: This master's current role is: LEADER
I20260927 14:55:45.634426 21489 sys_catalog.cc:455] T 00000000000000000000000000000000 P 353630ae8f054bb8ba359f332561f032 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "353630ae8f054bb8ba359f332561f032" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "353630ae8f054bb8ba359f332561f032" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46217 } } }
I20260927 14:55:45.634578 21489 sys_catalog.cc:458] T 00000000000000000000000000000000 P 353630ae8f054bb8ba359f332561f032 [sys.catalog]: This master's current role is: LEADER
W20260927 14:55:45.638447 21509 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 353630ae8f054bb8ba359f332561f032: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 14:55:45.638526 21509 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 14:55:45.638695 21498 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 14:55:45.639434 21486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:55:45.633630 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.55:58359 (local address 127.0.0.1:46217)
0927 14:55:45.633749 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0927 14:55:45.633753 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 14:55:45.635134 (+ 1381us) server_negotiation.cc:407] Connection header received
0927 14:55:45.635461 (+ 327us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:55:45.635465 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:55:45.635513 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:55:45.635595 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:55:45.635972 (+ 377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:55:45.636469 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:55:45.637560 (+ 1091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:55:45.637892 (+ 332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:55:45.638009 (+ 117us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:55:45.638954 (+ 945us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:55:45.638957 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:55:45.638960 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 14:55:45.638973 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:55:45.639005 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 14:55:45.639017 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:55:45.639111 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0927 14:55:45.639289 (+ 178us) server_negotiation.cc:299] Negotiation successful
0927 14:55:45.639320 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260927 14:55:45.639642 21498 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 14:55:45.644213 21498 catalog_manager.cc:1347] Generated new cluster ID: 0992bd3a65fe48f1bb5a287879679d93
I20260927 14:55:45.644274 21498 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 14:55:45.652581 21498 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 14:55:45.653692 21498 catalog_manager.cc:1504] Loading token signing keys...
I20260927 14:55:45.663962 21498 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 353630ae8f054bb8ba359f332561f032: Generated new TSK 0
I20260927 14:55:45.664788 21498 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 14:55:45.675983 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6936m5/build/debug/bin/kudu with pid 21415
W20260927 14:55:45.681178 21400 connection.cc:535] client connection to 127.0.0.1:46217 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:46217): Connection reset by peer (error 104)
2026-09-27T14:55:45Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1404 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-task6936m5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790520684602251-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 14:55:45 dist-test-slave-xk1g krb5kdc[21514](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 14:55:45 dist-test-slave-xk1g krb5kdc[21514](info): set up 2 sockets
Sep 27 14:55:45 dist-test-slave-xk1g krb5kdc[21514](info): commencing operation
krb5kdc: starting...
W20260927 14:55:46.801250 392 mini_kdc.cc:121] Time spent starting KDC: real 1.106s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6936m5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790520684602251-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6936m5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790520684602251-392-0/krb5kdc/test-user.keytab.
Sep 27 14:55:46 dist-test-slave-xk1g krb5kdc[21514](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790520946, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T14:55:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T14:55:46Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6936m5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790520684602251-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6936m5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790520684602251-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 14:55:46.919812 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6936m5/build/debug/bin/kudu
/tmp/dist-test-task6936m5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790520684602251-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790520684602251-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790520684602251-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790520684602251-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35129
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790520684602251-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:41065
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35129
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790520684602251-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790520684602251-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790520684602251-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 14:55:47.029953 21527 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 14:55:47.030218 21527 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 14:55:47.030259 21527 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 14:55:47.034204 21527 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 14:55:47.034266 21527 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 14:55:47.034286 21527 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 14:55:47.034302 21527 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 14:55:47.034318 21527 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 14:55:47.039431 21527 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41065
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790520684602251-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790520684602251-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35129
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790520684602251-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35129
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790520684602251-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21527
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790520684602251-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:15:17 UTC on ccycloud.quasar-ledxfe.root.comops.site
build id 4676134
I20260927 14:55:47.040397 21527 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 14:55:47.041419 21527 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 14:55:47.049901 21532 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 14:55:47.049886 21533 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 14:55:47.049996 21527 server_base.cc:1034] running on GCE node
W20260927 14:55:47.049901 21535 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 14:55:47.050755 21527 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 14:55:47.051797 21527 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 14:55:47.052960 21527 hybrid_clock.cc:648] HybridClock initialized: now 1790520947052929 us; error 43 us; skew 500 ppm
Sep 27 14:55:47 dist-test-slave-xk1g krb5kdc[21514](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790520947, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 14:55:47.055697 21527 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 14:55:47.056574 21527 webserver.cc:460] Webserver started at http://127.0.0.1:43649/ using document root <none> and password file <none>
I20260927 14:55:47.057193 21527 fs_manager.cc:359] Metadata directory not provided
I20260927 14:55:47.057286 21527 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 14:55:47.057518 21527 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 14:55:47.059516 21527 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6936m5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790520684602251-392-0/minicluster-data/master-0/data/instance:
uuid: "ed79aef6004045c5806dfc1bb3b8fcac"
format_stamp: "Formatted at 2026-09-27 14:55:47 on dist-test-slave-xk1g"
server_key: "1b963225a5d35d21776511bc75c44532"
server_key_iv: "8ca5e4a5787b4a53e02c1eb8ceebe5b7"
server_key_version: "encryptionkey@0"
I20260927 14:55:47.060073 21527 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6936m5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790520684602251-392-0/minicluster-data/master-0/wal/instance:
uuid: "ed79aef6004045c5806dfc1bb3b8fcac"
format_stamp: "Formatted at 2026-09-27 14:55:47 on dist-test-slave-xk1g"
server_key: "1b963225a5d35d21776511bc75c44532"
server_key_iv: "8ca5e4a5787b4a53e02c1eb8ceebe5b7"
server_key_version: "encryptionkey@0"
I20260927 14:55:47.063732 21527 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.003s
I20260927 14:55:47.066079 21542 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 14:55:47.067237 21527 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 14:55:47.067384 21527 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790520684602251-392-0/minicluster-data/master-0/data,/tmp/dist-test-task6936m5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790520684602251-392-0/minicluster-data/master-0/wal
uuid: "ed79aef6004045c5806dfc1bb3b8fcac"
format_stamp: "Formatted at 2026-09-27 14:55:47 on dist-test-slave-xk1g"
server_key: "1b963225a5d35d21776511bc75c44532"
server_key_iv: "8ca5e4a5787b4a53e02c1eb8ceebe5b7"
server_key_version: "encryptionkey@0"
I20260927 14:55:47.067507 21527 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790520684602251-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790520684602251-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790520684602251-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 14:55:47.072057 21527 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 14:55:47.074805 21527 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 14:55:47.074996 21527 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 14:55:47.081987 21527 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35129
I20260927 14:55:47.082015 21594 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35129 every 8 connection(s)
I20260927 14:55:47.082854 21527 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6936m5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790520684602251-392-0/minicluster-data/master-0/data/info.pb
I20260927 14:55:47.085402 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6936m5/build/debug/bin/kudu as pid 21527
I20260927 14:55:47.085554 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6936m5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790520684602251-392-0/minicluster-data/master-0/wal/instance
I20260927 14:55:47.085659 21595 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 14:55:47.085862 392 external_mini_cluster.cc:1428] Setting key 31bc180f8ff9770b5d4f3b965fee6f18
I20260927 14:55:47.090705 21598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:55:47.087201 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.55:51061 (local address 127.0.0.1:35129)
0927 14:55:47.087611 (+ 410us) server_negotiation.cc:206] Beginning negotiation
0927 14:55:47.087625 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 14:55:47.087659 (+ 34us) server_negotiation.cc:407] Connection header received
0927 14:55:47.088234 (+ 575us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:55:47.088249 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:55:47.088517 (+ 268us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:55:47.088840 (+ 323us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:55:47.089581 (+ 741us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:55:47.089587 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:55:47.089601 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 14:55:47.089618 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:55:47.089667 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 14:55:47.089685 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:55:47.089832 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0927 14:55:47.089903 (+ 71us) server_negotiation.cc:299] Negotiation successful
0927 14:55:47.090054 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":279,"thread_start_us":132,"threads_started":1}
I20260927 14:55:47.091995 21595 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ed79aef6004045c5806dfc1bb3b8fcac: Bootstrap starting.
I20260927 14:55:47.093307 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 14:55:47.094312 21595 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ed79aef6004045c5806dfc1bb3b8fcac: Neither blocks nor log segments found. Creating new log.
I20260927 14:55:47.095293 21595 log.cc:824] T 00000000000000000000000000000000 P ed79aef6004045c5806dfc1bb3b8fcac: Log is configured to *not* fsync() on all Append() calls
I20260927 14:55:47.096798 21598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:55:47.095191 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.55:58333 (local address 127.0.0.1:35129)
0927 14:55:47.095297 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0927 14:55:47.095300 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 14:55:47.095545 (+ 245us) server_negotiation.cc:407] Connection header received
0927 14:55:47.095700 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:55:47.095702 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:55:47.095734 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:55:47.095820 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:55:47.096458 (+ 638us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:55:47.096460 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:55:47.096462 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 14:55:47.096472 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:55:47.096493 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 14:55:47.096502 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:55:47.096569 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0927 14:55:47.096664 (+ 95us) server_negotiation.cc:299] Negotiation successful
0927 14:55:47.096691 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260927 14:55:47.097555 21595 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ed79aef6004045c5806dfc1bb3b8fcac: No bootstrap required, opened a new log
I20260927 14:55:47.100777 21595 raft_consensus.cc:348] T 00000000000000000000000000000000 P ed79aef6004045c5806dfc1bb3b8fcac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed79aef6004045c5806dfc1bb3b8fcac" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35129 } }
I20260927 14:55:47.101002 21595 raft_consensus.cc:374] T 00000000000000000000000000000000 P ed79aef6004045c5806dfc1bb3b8fcac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 14:55:47.101076 21595 raft_consensus.cc:729] T 00000000000000000000000000000000 P ed79aef6004045c5806dfc1bb3b8fcac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ed79aef6004045c5806dfc1bb3b8fcac, State: Initialized, Role: FOLLOWER
I20260927 14:55:47.101500 21595 consensus_queue.cc:260] T 00000000000000000000000000000000 P ed79aef6004045c5806dfc1bb3b8fcac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed79aef6004045c5806dfc1bb3b8fcac" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35129 } }
I20260927 14:55:47.101655 21595 raft_consensus.cc:388] T 00000000000000000000000000000000 P ed79aef6004045c5806dfc1bb3b8fcac [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 14:55:47.101719 21595 raft_consensus.cc:482] T 00000000000000000000000000000000 P ed79aef6004045c5806dfc1bb3b8fcac [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 14:55:47.101810 21595 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ed79aef6004045c5806dfc1bb3b8fcac [term 0 FOLLOWER]: Advancing to term 1
I20260927 14:55:47.102610 21595 raft_consensus.cc:504] T 00000000000000000000000000000000 P ed79aef6004045c5806dfc1bb3b8fcac [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed79aef6004045c5806dfc1bb3b8fcac" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35129 } }
I20260927 14:55:47.102928 21595 leader_election.cc:302] T 00000000000000000000000000000000 P ed79aef6004045c5806dfc1bb3b8fcac [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ed79aef6004045c5806dfc1bb3b8fcac; no voters:
I20260927 14:55:47.103235 21595 leader_election.cc:288] T 00000000000000000000000000000000 P ed79aef6004045c5806dfc1bb3b8fcac [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 14:55:47.103331 21605 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ed79aef6004045c5806dfc1bb3b8fcac [term 1 FOLLOWER]: Leader election won for term 1
I20260927 14:55:47.103559 21605 raft_consensus.cc:686] T 00000000000000000000000000000000 P ed79aef6004045c5806dfc1bb3b8fcac [term 1 LEADER]: Becoming Leader. State: Replica: ed79aef6004045c5806dfc1bb3b8fcac, State: Running, Role: LEADER
I20260927 14:55:47.103900 21605 consensus_queue.cc:237] T 00000000000000000000000000000000 P ed79aef6004045c5806dfc1bb3b8fcac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed79aef6004045c5806dfc1bb3b8fcac" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35129 } }
I20260927 14:55:47.104434 21595 sys_catalog.cc:564] T 00000000000000000000000000000000 P ed79aef6004045c5806dfc1bb3b8fcac [sys.catalog]: configured and running, proceeding with master startup.
I20260927 14:55:47.105650 21606 sys_catalog.cc:455] T 00000000000000000000000000000000 P ed79aef6004045c5806dfc1bb3b8fcac [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ed79aef6004045c5806dfc1bb3b8fcac" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed79aef6004045c5806dfc1bb3b8fcac" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35129 } } }
I20260927 14:55:47.105664 21607 sys_catalog.cc:455] T 00000000000000000000000000000000 P ed79aef6004045c5806dfc1bb3b8fcac [sys.catalog]: SysCatalogTable state changed. Reason: New leader ed79aef6004045c5806dfc1bb3b8fcac. Latest consensus state: current_term: 1 leader_uuid: "ed79aef6004045c5806dfc1bb3b8fcac" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed79aef6004045c5806dfc1bb3b8fcac" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35129 } } }
I20260927 14:55:47.105794 21606 sys_catalog.cc:458] T 00000000000000000000000000000000 P ed79aef6004045c5806dfc1bb3b8fcac [sys.catalog]: This master's current role is: LEADER
I20260927 14:55:47.105795 21607 sys_catalog.cc:458] T 00000000000000000000000000000000 P ed79aef6004045c5806dfc1bb3b8fcac [sys.catalog]: This master's current role is: LEADER
I20260927 14:55:47.106145 21614 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 14:55:47.109143 21614 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 14:55:47.114351 21614 catalog_manager.cc:1347] Generated new cluster ID: beefa3fcf42e4898ae2bb312758549dd
I20260927 14:55:47.114423 21614 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 14:55:47.123826 21614 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 14:55:47.124109 21614 catalog_manager.cc:1504] Loading token signing keys...
I20260927 14:55:47.135028 21614 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ed79aef6004045c5806dfc1bb3b8fcac: Generated new TSK 0
I20260927 14:55:47.135807 21614 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 14:55:47.139923 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6936m5/build/debug/bin/kudu with pid 21527
2026-09-27T14:55:47Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1463 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-task6936m5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790520684602251-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 14:55:47 dist-test-slave-xk1g krb5kdc[21626](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 14:55:47 dist-test-slave-xk1g krb5kdc[21626](info): set up 2 sockets
Sep 27 14:55:47 dist-test-slave-xk1g krb5kdc[21626](info): commencing operation
krb5kdc: starting...
W20260927 14:55:48.185602 392 mini_kdc.cc:121] Time spent starting KDC: real 1.026s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6936m5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790520684602251-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6936m5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790520684602251-392-0/krb5kdc/test-user.keytab.
Sep 27 14:55:48 dist-test-slave-xk1g krb5kdc[21626](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790520948, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T14:55:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T14:55:48Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6936m5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790520684602251-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6936m5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790520684602251-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 14:55:48.304960 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6936m5/build/debug/bin/kudu
/tmp/dist-test-task6936m5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790520684602251-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790520684602251-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790520684602251-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790520684602251-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34387
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790520684602251-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38975
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34387
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790520684602251-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790520684602251-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790520684602251-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 14:55:48.417249 21639 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 14:55:48.417577 21639 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 14:55:48.417668 21639 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 14:55:48.421869 21639 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 14:55:48.421963 21639 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 14:55:48.422001 21639 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 14:55:48.422057 21639 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 14:55:48.422096 21639 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 14:55:48.427474 21639 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38975
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790520684602251-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790520684602251-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34387
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790520684602251-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34387
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790520684602251-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21639
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790520684602251-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:15:17 UTC on ccycloud.quasar-ledxfe.root.comops.site
build id 4676134
I20260927 14:55:48.428601 21639 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 14:55:48.429718 21639 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 14:55:48.438130 21644 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 14:55:48.438148 21647 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 14:55:48.438243 21639 server_base.cc:1034] running on GCE node
W20260927 14:55:48.438176 21645 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 14:55:48.439105 21639 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 14:55:48.440198 21639 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 14:55:48.441385 21639 hybrid_clock.cc:648] HybridClock initialized: now 1790520948441369 us; error 26 us; skew 500 ppm
Sep 27 14:55:48 dist-test-slave-xk1g krb5kdc[21626](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790520948, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 14:55:48.444165 21639 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 14:55:48.445142 21639 webserver.cc:460] Webserver started at http://127.0.0.1:41697/ using document root <none> and password file <none>
I20260927 14:55:48.445678 21639 fs_manager.cc:359] Metadata directory not provided
I20260927 14:55:48.445735 21639 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 14:55:48.445894 21639 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 14:55:48.447958 21639 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6936m5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790520684602251-392-0/minicluster-data/master-0/data/instance:
uuid: "60f31fb9655d4e85b256bac39e34db27"
format_stamp: "Formatted at 2026-09-27 14:55:48 on dist-test-slave-xk1g"
server_key: "35ce1f0c178292f0380466e70895eb81"
server_key_iv: "fc205f1019507baedf184e801540031f"
server_key_version: "encryptionkey@0"
I20260927 14:55:48.448501 21639 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6936m5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790520684602251-392-0/minicluster-data/master-0/wal/instance:
uuid: "60f31fb9655d4e85b256bac39e34db27"
format_stamp: "Formatted at 2026-09-27 14:55:48 on dist-test-slave-xk1g"
server_key: "35ce1f0c178292f0380466e70895eb81"
server_key_iv: "fc205f1019507baedf184e801540031f"
server_key_version: "encryptionkey@0"
I20260927 14:55:48.452186 21639 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260927 14:55:48.454739 21654 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 14:55:48.455974 21639 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 14:55:48.456080 21639 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790520684602251-392-0/minicluster-data/master-0/data,/tmp/dist-test-task6936m5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790520684602251-392-0/minicluster-data/master-0/wal
uuid: "60f31fb9655d4e85b256bac39e34db27"
format_stamp: "Formatted at 2026-09-27 14:55:48 on dist-test-slave-xk1g"
server_key: "35ce1f0c178292f0380466e70895eb81"
server_key_iv: "fc205f1019507baedf184e801540031f"
server_key_version: "encryptionkey@0"
I20260927 14:55:48.456168 21639 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790520684602251-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790520684602251-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790520684602251-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 14:55:48.473868 21639 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 14:55:48.477015 21639 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 14:55:48.477193 21639 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 14:55:48.484163 21639 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34387
I20260927 14:55:48.484182 21706 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34387 every 8 connection(s)
I20260927 14:55:48.485150 21639 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6936m5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790520684602251-392-0/minicluster-data/master-0/data/info.pb
I20260927 14:55:48.487769 21707 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 14:55:48.491667 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6936m5/build/debug/bin/kudu as pid 21639
I20260927 14:55:48.491842 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6936m5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790520684602251-392-0/minicluster-data/master-0/wal/instance
I20260927 14:55:48.492132 392 external_mini_cluster.cc:1428] Setting key 1fe435263da8b8da122e4ccd22bfc1ab
I20260927 14:55:48.493739 21707 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 60f31fb9655d4e85b256bac39e34db27: Bootstrap starting.
I20260927 14:55:48.495971 21707 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 60f31fb9655d4e85b256bac39e34db27: Neither blocks nor log segments found. Creating new log.
I20260927 14:55:48.496703 21707 log.cc:824] T 00000000000000000000000000000000 P 60f31fb9655d4e85b256bac39e34db27: Log is configured to *not* fsync() on all Append() calls
I20260927 14:55:48.498485 21707 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 60f31fb9655d4e85b256bac39e34db27: No bootstrap required, opened a new log
Sep 27 14:55:48 dist-test-slave-xk1g krb5kdc[21626](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790520948, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260927 14:55:48.502156 21707 raft_consensus.cc:348] T 00000000000000000000000000000000 P 60f31fb9655d4e85b256bac39e34db27 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "60f31fb9655d4e85b256bac39e34db27" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34387 } }
I20260927 14:55:48.502419 21707 raft_consensus.cc:374] T 00000000000000000000000000000000 P 60f31fb9655d4e85b256bac39e34db27 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 14:55:48.502537 21707 raft_consensus.cc:729] T 00000000000000000000000000000000 P 60f31fb9655d4e85b256bac39e34db27 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60f31fb9655d4e85b256bac39e34db27, State: Initialized, Role: FOLLOWER
I20260927 14:55:48.503026 21707 consensus_queue.cc:260] T 00000000000000000000000000000000 P 60f31fb9655d4e85b256bac39e34db27 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "60f31fb9655d4e85b256bac39e34db27" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34387 } }
I20260927 14:55:48.503208 21707 raft_consensus.cc:388] T 00000000000000000000000000000000 P 60f31fb9655d4e85b256bac39e34db27 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 14:55:48.503296 21707 raft_consensus.cc:482] T 00000000000000000000000000000000 P 60f31fb9655d4e85b256bac39e34db27 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 14:55:48.503413 21707 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 60f31fb9655d4e85b256bac39e34db27 [term 0 FOLLOWER]: Advancing to term 1
I20260927 14:55:48.504572 21707 raft_consensus.cc:504] T 00000000000000000000000000000000 P 60f31fb9655d4e85b256bac39e34db27 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "60f31fb9655d4e85b256bac39e34db27" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34387 } }
I20260927 14:55:48.504910 21707 leader_election.cc:302] T 00000000000000000000000000000000 P 60f31fb9655d4e85b256bac39e34db27 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 60f31fb9655d4e85b256bac39e34db27; no voters:
I20260927 14:55:48.505245 21707 leader_election.cc:288] T 00000000000000000000000000000000 P 60f31fb9655d4e85b256bac39e34db27 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 14:55:48.505345 21712 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 60f31fb9655d4e85b256bac39e34db27 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 14:55:48.505818 21712 raft_consensus.cc:686] T 00000000000000000000000000000000 P 60f31fb9655d4e85b256bac39e34db27 [term 1 LEADER]: Becoming Leader. State: Replica: 60f31fb9655d4e85b256bac39e34db27, State: Running, Role: LEADER
I20260927 14:55:48.506171 21712 consensus_queue.cc:237] T 00000000000000000000000000000000 P 60f31fb9655d4e85b256bac39e34db27 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "60f31fb9655d4e85b256bac39e34db27" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34387 } }
I20260927 14:55:48.506557 21707 sys_catalog.cc:564] T 00000000000000000000000000000000 P 60f31fb9655d4e85b256bac39e34db27 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 14:55:48.506901 21710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:55:48.493567 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.55:58199 (local address 127.0.0.1:34387)
0927 14:55:48.494048 (+ 481us) server_negotiation.cc:206] Beginning negotiation
0927 14:55:48.494064 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0927 14:55:48.494100 (+ 36us) server_negotiation.cc:407] Connection header received
0927 14:55:48.494728 (+ 628us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:55:48.494744 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:55:48.495109 (+ 365us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:55:48.495455 (+ 346us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:55:48.496307 (+ 852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:55:48.497412 (+ 1105us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:55:48.498286 (+ 874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:55:48.498805 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:55:48.498981 (+ 176us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:55:48.501517 (+ 2536us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:55:48.501523 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:55:48.501535 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 14:55:48.501561 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:55:48.503434 (+ 1873us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:55:48.504010 (+ 576us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:55:48.504015 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:55:48.504018 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:55:48.504143 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:55:48.504541 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:55:48.504544 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:55:48.504545 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:55:48.505002 (+ 457us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:55:48.505294 (+ 292us) server_negotiation.cc:1035] Waiting for connection context
0927 14:55:48.505608 (+ 314us) server_negotiation.cc:299] Negotiation successful
0927 14:55:48.505871 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":343,"thread_start_us":149,"threads_started":1}
I20260927 14:55:48.507772 21713 sys_catalog.cc:455] T 00000000000000000000000000000000 P 60f31fb9655d4e85b256bac39e34db27 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 60f31fb9655d4e85b256bac39e34db27. Latest consensus state: current_term: 1 leader_uuid: "60f31fb9655d4e85b256bac39e34db27" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "60f31fb9655d4e85b256bac39e34db27" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34387 } } }
I20260927 14:55:48.507899 21713 sys_catalog.cc:458] T 00000000000000000000000000000000 P 60f31fb9655d4e85b256bac39e34db27 [sys.catalog]: This master's current role is: LEADER
I20260927 14:55:48.508301 21714 sys_catalog.cc:455] T 00000000000000000000000000000000 P 60f31fb9655d4e85b256bac39e34db27 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "60f31fb9655d4e85b256bac39e34db27" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "60f31fb9655d4e85b256bac39e34db27" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34387 } } }
I20260927 14:55:48.508455 21714 sys_catalog.cc:458] T 00000000000000000000000000000000 P 60f31fb9655d4e85b256bac39e34db27 [sys.catalog]: This master's current role is: LEADER
I20260927 14:55:48.510248 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
W20260927 14:55:48.511667 21727 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 60f31fb9655d4e85b256bac39e34db27: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 14:55:48.511757 21727 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 14:55:48.511857 21722 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 14:55:48.512707 21722 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 14:55:48.518757 21722 catalog_manager.cc:1347] Generated new cluster ID: bd64dae02a3c4d338a4e2652a06bb2d0
I20260927 14:55:48.518899 21722 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 14:55:48.522411 21710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:55:48.513524 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.55:55681 (local address 127.0.0.1:34387)
0927 14:55:48.513647 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0927 14:55:48.513652 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 14:55:48.513894 (+ 242us) server_negotiation.cc:407] Connection header received
0927 14:55:48.514058 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:55:48.514061 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:55:48.514100 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:55:48.514185 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:55:48.514937 (+ 752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:55:48.515819 (+ 882us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:55:48.516787 (+ 968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:55:48.517049 (+ 262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:55:48.517197 (+ 148us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:55:48.518960 (+ 1763us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:55:48.518963 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:55:48.518966 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 14:55:48.518988 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:55:48.520633 (+ 1645us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:55:48.521170 (+ 537us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:55:48.521173 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:55:48.521175 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:55:48.521254 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:55:48.521593 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:55:48.521596 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:55:48.521597 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:55:48.521780 (+ 183us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:55:48.521899 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0927 14:55:48.522183 (+ 284us) server_negotiation.cc:299] Negotiation successful
0927 14:55:48.522269 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260927 14:55:48.536337 21722 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 14:55:48.537647 21722 catalog_manager.cc:1504] Loading token signing keys...
I20260927 14:55:48.550006 21722 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 60f31fb9655d4e85b256bac39e34db27: Generated new TSK 0
I20260927 14:55:48.550776 21722 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 14:55:48.560454 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6936m5/build/debug/bin/kudu with pid 21639
W20260927 14:55:48.565902 21624 connection.cc:535] client connection to 127.0.0.1:34387 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:34387): Connection reset by peer (error 104)
2026-09-27T14:55:48Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (1420 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-task6936m5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790520684602251-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 14:55:48 dist-test-slave-xk1g krb5kdc[21738](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 14:55:48 dist-test-slave-xk1g krb5kdc[21738](info): set up 2 sockets
Sep 27 14:55:48 dist-test-slave-xk1g krb5kdc[21738](info): commencing operation
krb5kdc: starting...
W20260927 14:55:49.613246 392 mini_kdc.cc:121] Time spent starting KDC: real 1.033s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6936m5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790520684602251-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6936m5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790520684602251-392-0/krb5kdc/test-user.keytab.
Sep 27 14:55:49 dist-test-slave-xk1g krb5kdc[21738](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790520949, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T14:55:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T14:55:49Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6936m5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790520684602251-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6936m5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790520684602251-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 14:55:49.732896 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6936m5/build/debug/bin/kudu
/tmp/dist-test-task6936m5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790520684602251-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790520684602251-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790520684602251-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790520684602251-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38573
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790520684602251-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36279
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38573
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790520684602251-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790520684602251-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790520684602251-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 14:55:49.839388 21751 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 14:55:49.839700 21751 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 14:55:49.839790 21751 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 14:55:49.843822 21751 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 14:55:49.843918 21751 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 14:55:49.843954 21751 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 14:55:49.843995 21751 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 14:55:49.844027 21751 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 14:55:49.849339 21751 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36279
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790520684602251-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790520684602251-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38573
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790520684602251-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38573
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790520684602251-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21751
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790520684602251-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:15:17 UTC on ccycloud.quasar-ledxfe.root.comops.site
build id 4676134
I20260927 14:55:49.850400 21751 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 14:55:49.851428 21751 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 14:55:49.859978 21757 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 14:55:49.859985 21759 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 14:55:49.860004 21756 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 14:55:49.860411 21751 server_base.cc:1034] running on GCE node
I20260927 14:55:49.860976 21751 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 14:55:49.862054 21751 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 14:55:49.863242 21751 hybrid_clock.cc:648] HybridClock initialized: now 1790520949863229 us; error 25 us; skew 500 ppm
Sep 27 14:55:49 dist-test-slave-xk1g krb5kdc[21738](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790520949, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 14:55:49.865857 21751 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 14:55:49.866688 21751 webserver.cc:460] Webserver started at http://127.0.0.1:36491/ using document root <none> and password file <none>
I20260927 14:55:49.867237 21751 fs_manager.cc:359] Metadata directory not provided
I20260927 14:55:49.867327 21751 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 14:55:49.867563 21751 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 14:55:49.869619 21751 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6936m5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790520684602251-392-0/minicluster-data/master-0/data/instance:
uuid: "de0c3a8d600e4a6f95389c38d266478c"
format_stamp: "Formatted at 2026-09-27 14:55:49 on dist-test-slave-xk1g"
server_key: "20c8c16f29c49837c5bfb2983569097e"
server_key_iv: "0511766d4e30068c2907fa75d3492433"
server_key_version: "encryptionkey@0"
I20260927 14:55:49.870155 21751 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6936m5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790520684602251-392-0/minicluster-data/master-0/wal/instance:
uuid: "de0c3a8d600e4a6f95389c38d266478c"
format_stamp: "Formatted at 2026-09-27 14:55:49 on dist-test-slave-xk1g"
server_key: "20c8c16f29c49837c5bfb2983569097e"
server_key_iv: "0511766d4e30068c2907fa75d3492433"
server_key_version: "encryptionkey@0"
I20260927 14:55:49.873718 21751 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260927 14:55:49.875937 21766 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 14:55:49.877143 21751 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 14:55:49.877266 21751 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790520684602251-392-0/minicluster-data/master-0/data,/tmp/dist-test-task6936m5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790520684602251-392-0/minicluster-data/master-0/wal
uuid: "de0c3a8d600e4a6f95389c38d266478c"
format_stamp: "Formatted at 2026-09-27 14:55:49 on dist-test-slave-xk1g"
server_key: "20c8c16f29c49837c5bfb2983569097e"
server_key_iv: "0511766d4e30068c2907fa75d3492433"
server_key_version: "encryptionkey@0"
I20260927 14:55:49.877381 21751 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790520684602251-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790520684602251-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790520684602251-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 14:55:49.897495 21751 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 14:55:49.900115 21751 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 14:55:49.900316 21751 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 14:55:49.907311 21751 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38573
I20260927 14:55:49.907334 21818 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38573 every 8 connection(s)
I20260927 14:55:49.908264 21751 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6936m5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790520684602251-392-0/minicluster-data/master-0/data/info.pb
I20260927 14:55:49.908619 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6936m5/build/debug/bin/kudu as pid 21751
I20260927 14:55:49.908782 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6936m5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790520684602251-392-0/minicluster-data/master-0/wal/instance
I20260927 14:55:49.909047 392 external_mini_cluster.cc:1428] Setting key 0ae2eb4503eeb21def9598b21f432354
I20260927 14:55:49.911098 21819 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 14:55:49.916474 21819 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P de0c3a8d600e4a6f95389c38d266478c: Bootstrap starting.
I20260927 14:55:49.917868 21822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:55:49.910475 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.55:37701 (local address 127.0.0.1:38573)
0927 14:55:49.910883 (+ 408us) server_negotiation.cc:206] Beginning negotiation
0927 14:55:49.910899 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0927 14:55:49.910937 (+ 38us) server_negotiation.cc:407] Connection header received
0927 14:55:49.911600 (+ 663us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:55:49.911618 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:55:49.911962 (+ 344us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:55:49.912298 (+ 336us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:55:49.912691 (+ 393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:55:49.913735 (+ 1044us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:55:49.914635 (+ 900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:55:49.915116 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:55:49.915288 (+ 172us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:55:49.916193 (+ 905us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:55:49.916197 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:55:49.916210 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 14:55:49.916630 (+ 420us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:55:49.916685 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 14:55:49.916707 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:55:49.916845 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0927 14:55:49.917034 (+ 189us) server_negotiation.cc:299] Negotiation successful
0927 14:55:49.917195 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":269,"thread_start_us":138,"threads_started":1}
I20260927 14:55:49.918793 21819 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P de0c3a8d600e4a6f95389c38d266478c: Neither blocks nor log segments found. Creating new log.
I20260927 14:55:49.919499 21819 log.cc:824] T 00000000000000000000000000000000 P de0c3a8d600e4a6f95389c38d266478c: Log is configured to *not* fsync() on all Append() calls
I20260927 14:55:49.920696 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 14:55:49.921586 21819 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P de0c3a8d600e4a6f95389c38d266478c: No bootstrap required, opened a new log
I20260927 14:55:49.926081 21819 raft_consensus.cc:348] T 00000000000000000000000000000000 P de0c3a8d600e4a6f95389c38d266478c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de0c3a8d600e4a6f95389c38d266478c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38573 } }
I20260927 14:55:49.926402 21819 raft_consensus.cc:374] T 00000000000000000000000000000000 P de0c3a8d600e4a6f95389c38d266478c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 14:55:49.926496 21819 raft_consensus.cc:729] T 00000000000000000000000000000000 P de0c3a8d600e4a6f95389c38d266478c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: de0c3a8d600e4a6f95389c38d266478c, State: Initialized, Role: FOLLOWER
I20260927 14:55:49.926905 21822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:55:49.922657 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.55:38431 (local address 127.0.0.1:38573)
0927 14:55:49.922778 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0927 14:55:49.922782 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 14:55:49.922979 (+ 197us) server_negotiation.cc:407] Connection header received
0927 14:55:49.923086 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:55:49.923088 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:55:49.923125 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:55:49.923187 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:55:49.923485 (+ 298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:55:49.923968 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:55:49.925227 (+ 1259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:55:49.925425 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:55:49.925502 (+ 77us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:55:49.926391 (+ 889us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:55:49.926396 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:55:49.926399 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 14:55:49.926415 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:55:49.926449 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 14:55:49.926461 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:55:49.926568 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0927 14:55:49.926715 (+ 147us) server_negotiation.cc:299] Negotiation successful
0927 14:55:49.926752 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260927 14:55:49.927029 21819 consensus_queue.cc:260] T 00000000000000000000000000000000 P de0c3a8d600e4a6f95389c38d266478c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de0c3a8d600e4a6f95389c38d266478c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38573 } }
I20260927 14:55:49.927209 21819 raft_consensus.cc:388] T 00000000000000000000000000000000 P de0c3a8d600e4a6f95389c38d266478c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 14:55:49.927284 21819 raft_consensus.cc:482] T 00000000000000000000000000000000 P de0c3a8d600e4a6f95389c38d266478c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 14:55:49.927386 21819 raft_consensus.cc:3037] T 00000000000000000000000000000000 P de0c3a8d600e4a6f95389c38d266478c [term 0 FOLLOWER]: Advancing to term 1
I20260927 14:55:49.928319 21819 raft_consensus.cc:504] T 00000000000000000000000000000000 P de0c3a8d600e4a6f95389c38d266478c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de0c3a8d600e4a6f95389c38d266478c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38573 } }
I20260927 14:55:49.928668 21819 leader_election.cc:302] T 00000000000000000000000000000000 P de0c3a8d600e4a6f95389c38d266478c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: de0c3a8d600e4a6f95389c38d266478c; no voters:
I20260927 14:55:49.929001 21819 leader_election.cc:288] T 00000000000000000000000000000000 P de0c3a8d600e4a6f95389c38d266478c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 14:55:49.929112 21829 raft_consensus.cc:2781] T 00000000000000000000000000000000 P de0c3a8d600e4a6f95389c38d266478c [term 1 FOLLOWER]: Leader election won for term 1
I20260927 14:55:49.929358 21829 raft_consensus.cc:686] T 00000000000000000000000000000000 P de0c3a8d600e4a6f95389c38d266478c [term 1 LEADER]: Becoming Leader. State: Replica: de0c3a8d600e4a6f95389c38d266478c, State: Running, Role: LEADER
I20260927 14:55:49.929656 21829 consensus_queue.cc:237] T 00000000000000000000000000000000 P de0c3a8d600e4a6f95389c38d266478c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de0c3a8d600e4a6f95389c38d266478c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38573 } }
I20260927 14:55:49.930336 21819 sys_catalog.cc:564] T 00000000000000000000000000000000 P de0c3a8d600e4a6f95389c38d266478c [sys.catalog]: configured and running, proceeding with master startup.
I20260927 14:55:49.931367 21830 sys_catalog.cc:455] T 00000000000000000000000000000000 P de0c3a8d600e4a6f95389c38d266478c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "de0c3a8d600e4a6f95389c38d266478c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de0c3a8d600e4a6f95389c38d266478c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38573 } } }
I20260927 14:55:49.931521 21830 sys_catalog.cc:458] T 00000000000000000000000000000000 P de0c3a8d600e4a6f95389c38d266478c [sys.catalog]: This master's current role is: LEADER
I20260927 14:55:49.931815 21831 sys_catalog.cc:455] T 00000000000000000000000000000000 P de0c3a8d600e4a6f95389c38d266478c [sys.catalog]: SysCatalogTable state changed. Reason: New leader de0c3a8d600e4a6f95389c38d266478c. Latest consensus state: current_term: 1 leader_uuid: "de0c3a8d600e4a6f95389c38d266478c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de0c3a8d600e4a6f95389c38d266478c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38573 } } }
I20260927 14:55:49.931921 21831 sys_catalog.cc:458] T 00000000000000000000000000000000 P de0c3a8d600e4a6f95389c38d266478c [sys.catalog]: This master's current role is: LEADER
I20260927 14:55:49.932013 21838 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 14:55:49.934873 21838 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 14:55:49.940658 21838 catalog_manager.cc:1347] Generated new cluster ID: b71cb883def34c23a4b0b7be58794967
I20260927 14:55:49.940724 21838 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 14:55:49.951239 21838 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 14:55:49.952124 21838 catalog_manager.cc:1504] Loading token signing keys...
I20260927 14:55:49.959729 21838 catalog_manager.cc:5868] T 00000000000000000000000000000000 P de0c3a8d600e4a6f95389c38d266478c: Generated new TSK 0
I20260927 14:55:49.960546 21838 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 14:55:49.975134 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6936m5/build/debug/bin/kudu with pid 21751
W20260927 14:55:49.980419 21736 connection.cc:535] client connection to 127.0.0.1:38573 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:38573): Connection reset by peer (error 104)
2026-09-27T14:55:49Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1414 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-task6936m5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790520684602251-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 14:55:50 dist-test-slave-xk1g krb5kdc[21850](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 14:55:50 dist-test-slave-xk1g krb5kdc[21850](info): set up 2 sockets
Sep 27 14:55:50 dist-test-slave-xk1g krb5kdc[21850](info): commencing operation
krb5kdc: starting...
W20260927 14:55:51.005272 392 mini_kdc.cc:121] Time spent starting KDC: real 1.010s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6936m5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790520684602251-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6936m5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790520684602251-392-0/krb5kdc/test-user.keytab.
Sep 27 14:55:51 dist-test-slave-xk1g krb5kdc[21850](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790520951, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T14:55:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T14:55:51Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6936m5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790520684602251-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6936m5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790520684602251-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 14:55:51.123521 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6936m5/build/debug/bin/kudu
/tmp/dist-test-task6936m5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790520684602251-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790520684602251-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790520684602251-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790520684602251-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38865
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790520684602251-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35539
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38865
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790520684602251-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790520684602251-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790520684602251-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 14:55:51.233937 21863 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 14:55:51.234198 21863 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 14:55:51.234274 21863 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 14:55:51.238268 21863 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 14:55:51.238336 21863 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 14:55:51.238355 21863 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 14:55:51.238415 21863 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 14:55:51.238444 21863 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 14:55:51.243747 21863 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35539
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790520684602251-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790520684602251-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38865
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790520684602251-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38865
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790520684602251-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21863
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790520684602251-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:15:17 UTC on ccycloud.quasar-ledxfe.root.comops.site
build id 4676134
I20260927 14:55:51.244813 21863 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 14:55:51.245838 21863 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 14:55:51.254297 21868 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 14:55:51.254297 21871 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 14:55:51.254314 21869 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 14:55:51.254818 21863 server_base.cc:1034] running on GCE node
I20260927 14:55:51.255481 21863 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 14:55:51.256644 21863 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 14:55:51.257887 21863 hybrid_clock.cc:648] HybridClock initialized: now 1790520951257854 us; error 49 us; skew 500 ppm
Sep 27 14:55:51 dist-test-slave-xk1g krb5kdc[21850](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790520951, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 14:55:51.264309 21863 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 14:55:51.265430 21863 webserver.cc:460] Webserver started at http://127.0.0.1:46681/ using document root <none> and password file <none>
I20260927 14:55:51.266024 21863 fs_manager.cc:359] Metadata directory not provided
I20260927 14:55:51.266124 21863 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 14:55:51.266371 21863 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 14:55:51.268484 21863 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6936m5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790520684602251-392-0/minicluster-data/master-0/data/instance:
uuid: "3fa11c759c234d98bee4c52d6e42c24b"
format_stamp: "Formatted at 2026-09-27 14:55:51 on dist-test-slave-xk1g"
server_key: "48989a9e2a2bb1b7788bb5d2165b525e"
server_key_iv: "251861eeeee6e2686215f2c69f720ead"
server_key_version: "encryptionkey@0"
I20260927 14:55:51.269163 21863 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6936m5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790520684602251-392-0/minicluster-data/master-0/wal/instance:
uuid: "3fa11c759c234d98bee4c52d6e42c24b"
format_stamp: "Formatted at 2026-09-27 14:55:51 on dist-test-slave-xk1g"
server_key: "48989a9e2a2bb1b7788bb5d2165b525e"
server_key_iv: "251861eeeee6e2686215f2c69f720ead"
server_key_version: "encryptionkey@0"
I20260927 14:55:51.272974 21863 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260927 14:55:51.275403 21878 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 14:55:51.276624 21863 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 14:55:51.276774 21863 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790520684602251-392-0/minicluster-data/master-0/data,/tmp/dist-test-task6936m5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790520684602251-392-0/minicluster-data/master-0/wal
uuid: "3fa11c759c234d98bee4c52d6e42c24b"
format_stamp: "Formatted at 2026-09-27 14:55:51 on dist-test-slave-xk1g"
server_key: "48989a9e2a2bb1b7788bb5d2165b525e"
server_key_iv: "251861eeeee6e2686215f2c69f720ead"
server_key_version: "encryptionkey@0"
I20260927 14:55:51.276898 21863 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790520684602251-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790520684602251-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790520684602251-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 14:55:51.281390 21863 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 14:55:51.284292 21863 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 14:55:51.284511 21863 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 14:55:51.292245 21863 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38865
I20260927 14:55:51.292274 21930 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38865 every 8 connection(s)
I20260927 14:55:51.293212 21863 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6936m5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790520684602251-392-0/minicluster-data/master-0/data/info.pb
I20260927 14:55:51.295789 21931 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 14:55:51.299173 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6936m5/build/debug/bin/kudu as pid 21863
I20260927 14:55:51.299314 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6936m5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790520684602251-392-0/minicluster-data/master-0/wal/instance
I20260927 14:55:51.299584 392 external_mini_cluster.cc:1428] Setting key 62b2b0b400019b9d52a19ff83c717874
I20260927 14:55:51.301702 21931 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3fa11c759c234d98bee4c52d6e42c24b: Bootstrap starting.
I20260927 14:55:51.303951 21931 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3fa11c759c234d98bee4c52d6e42c24b: Neither blocks nor log segments found. Creating new log.
I20260927 14:55:51.304706 21931 log.cc:824] T 00000000000000000000000000000000 P 3fa11c759c234d98bee4c52d6e42c24b: Log is configured to *not* fsync() on all Append() calls
I20260927 14:55:51.304983 21934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:55:51.301036 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.55:57025 (local address 127.0.0.1:38865)
0927 14:55:51.301498 (+ 462us) server_negotiation.cc:206] Beginning negotiation
0927 14:55:51.301512 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 14:55:51.301546 (+ 34us) server_negotiation.cc:407] Connection header received
0927 14:55:51.302242 (+ 696us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:55:51.302259 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:55:51.302553 (+ 294us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:55:51.302886 (+ 333us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:55:51.303651 (+ 765us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:55:51.303655 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:55:51.303667 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 14:55:51.303677 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:55:51.303727 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 14:55:51.303746 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:55:51.303861 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0927 14:55:51.303961 (+ 100us) server_negotiation.cc:299] Negotiation successful
0927 14:55:51.304236 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":289,"thread_start_us":130,"threads_started":1}
I20260927 14:55:51.306553 21931 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3fa11c759c234d98bee4c52d6e42c24b: No bootstrap required, opened a new log
I20260927 14:55:51.307834 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 14:55:51.311517 21934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:55:51.310050 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.55:53535 (local address 127.0.0.1:38865)
0927 14:55:51.310166 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0927 14:55:51.310172 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 14:55:51.310304 (+ 132us) server_negotiation.cc:407] Connection header received
0927 14:55:51.310403 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:55:51.310405 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:55:51.310439 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:55:51.310537 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:55:51.311171 (+ 634us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:55:51.311174 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:55:51.311177 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 14:55:51.311188 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:55:51.311214 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 14:55:51.311227 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:55:51.311309 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0927 14:55:51.311375 (+ 66us) server_negotiation.cc:299] Negotiation successful
0927 14:55:51.311412 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260927 14:55:51.311297 21931 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3fa11c759c234d98bee4c52d6e42c24b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3fa11c759c234d98bee4c52d6e42c24b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38865 } }
I20260927 14:55:51.311606 21931 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3fa11c759c234d98bee4c52d6e42c24b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 14:55:51.311661 21931 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3fa11c759c234d98bee4c52d6e42c24b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3fa11c759c234d98bee4c52d6e42c24b, State: Initialized, Role: FOLLOWER
I20260927 14:55:51.312088 21931 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3fa11c759c234d98bee4c52d6e42c24b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3fa11c759c234d98bee4c52d6e42c24b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38865 } }
I20260927 14:55:51.312212 21931 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3fa11c759c234d98bee4c52d6e42c24b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 14:55:51.312291 21931 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3fa11c759c234d98bee4c52d6e42c24b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 14:55:51.312402 21931 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3fa11c759c234d98bee4c52d6e42c24b [term 0 FOLLOWER]: Advancing to term 1
I20260927 14:55:51.313372 21931 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3fa11c759c234d98bee4c52d6e42c24b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3fa11c759c234d98bee4c52d6e42c24b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38865 } }
I20260927 14:55:51.313687 21931 leader_election.cc:302] T 00000000000000000000000000000000 P 3fa11c759c234d98bee4c52d6e42c24b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3fa11c759c234d98bee4c52d6e42c24b; no voters:
I20260927 14:55:51.313939 21931 leader_election.cc:288] T 00000000000000000000000000000000 P 3fa11c759c234d98bee4c52d6e42c24b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 14:55:51.314054 21941 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3fa11c759c234d98bee4c52d6e42c24b [term 1 FOLLOWER]: Leader election won for term 1
I20260927 14:55:51.314270 21941 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3fa11c759c234d98bee4c52d6e42c24b [term 1 LEADER]: Becoming Leader. State: Replica: 3fa11c759c234d98bee4c52d6e42c24b, State: Running, Role: LEADER
I20260927 14:55:51.314574 21941 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3fa11c759c234d98bee4c52d6e42c24b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3fa11c759c234d98bee4c52d6e42c24b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38865 } }
I20260927 14:55:51.315181 21931 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3fa11c759c234d98bee4c52d6e42c24b [sys.catalog]: configured and running, proceeding with master startup.
I20260927 14:55:51.316272 21942 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3fa11c759c234d98bee4c52d6e42c24b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3fa11c759c234d98bee4c52d6e42c24b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3fa11c759c234d98bee4c52d6e42c24b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38865 } } }
I20260927 14:55:51.316408 21942 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3fa11c759c234d98bee4c52d6e42c24b [sys.catalog]: This master's current role is: LEADER
I20260927 14:55:51.316727 21943 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3fa11c759c234d98bee4c52d6e42c24b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3fa11c759c234d98bee4c52d6e42c24b. Latest consensus state: current_term: 1 leader_uuid: "3fa11c759c234d98bee4c52d6e42c24b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3fa11c759c234d98bee4c52d6e42c24b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38865 } } }
I20260927 14:55:51.316817 21943 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3fa11c759c234d98bee4c52d6e42c24b [sys.catalog]: This master's current role is: LEADER
I20260927 14:55:51.316900 21950 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 14:55:51.319661 21950 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 14:55:51.325373 21950 catalog_manager.cc:1347] Generated new cluster ID: bf4d1b26593542b181e31e90d29369d6
I20260927 14:55:51.325461 21950 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 14:55:51.336066 21950 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 14:55:51.336366 21950 catalog_manager.cc:1504] Loading token signing keys...
I20260927 14:55:51.341228 21950 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3fa11c759c234d98bee4c52d6e42c24b: Generated new TSK 0
I20260927 14:55:51.341900 21950 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 14:55:51.347754 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6936m5/build/debug/bin/kudu with pid 21863
2026-09-27T14:55:51Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1372 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-task6936m5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790520684602251-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 14:55:51 dist-test-slave-xk1g krb5kdc[21962](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 14:55:51 dist-test-slave-xk1g krb5kdc[21962](info): set up 2 sockets
Sep 27 14:55:51 dist-test-slave-xk1g krb5kdc[21962](info): commencing operation
krb5kdc: starting...
W20260927 14:55:52.381439 392 mini_kdc.cc:121] Time spent starting KDC: real 1.015s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6936m5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790520684602251-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6936m5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790520684602251-392-0/krb5kdc/test-user.keytab.
Sep 27 14:55:52 dist-test-slave-xk1g krb5kdc[21962](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790520952, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T14:55:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T14:55:52Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6936m5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790520684602251-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6936m5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790520684602251-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 14:55:52.499603 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6936m5/build/debug/bin/kudu
/tmp/dist-test-task6936m5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790520684602251-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790520684602251-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790520684602251-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790520684602251-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42493
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790520684602251-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41353
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:42493
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790520684602251-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790520684602251-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790520684602251-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 14:55:52.609985 21975 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 14:55:52.610245 21975 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 14:55:52.610329 21975 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 14:55:52.614615 21975 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 14:55:52.614696 21975 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 14:55:52.614763 21975 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 14:55:52.614805 21975 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 14:55:52.614847 21975 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 14:55:52.620713 21975 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41353
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790520684602251-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790520684602251-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:42493
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790520684602251-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42493
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790520684602251-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21975
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790520684602251-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:15:17 UTC on ccycloud.quasar-ledxfe.root.comops.site
build id 4676134
I20260927 14:55:52.621860 21975 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 14:55:52.622861 21975 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 14:55:52.631419 21983 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 14:55:52.631493 21975 server_base.cc:1034] running on GCE node
W20260927 14:55:52.631436 21981 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 14:55:52.631434 21980 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 14:55:52.632179 21975 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 14:55:52.633273 21975 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 14:55:52.634428 21975 hybrid_clock.cc:648] HybridClock initialized: now 1790520952634419 us; error 28 us; skew 500 ppm
Sep 27 14:55:52 dist-test-slave-xk1g krb5kdc[21962](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790520952, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 14:55:52.637048 21975 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 14:55:52.637868 21975 webserver.cc:460] Webserver started at http://127.0.98.62:44321/ using document root <none> and password file <none>
I20260927 14:55:52.638352 21975 fs_manager.cc:359] Metadata directory not provided
I20260927 14:55:52.638417 21975 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 14:55:52.638571 21975 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 14:55:52.640448 21975 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6936m5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790520684602251-392-0/minicluster-data/master-0/data/instance:
uuid: "9a66b153d01345e4a8ff9aa8a117eb76"
format_stamp: "Formatted at 2026-09-27 14:55:52 on dist-test-slave-xk1g"
server_key: "51b68d69baa1760bc14997b2ca7b8dc5"
server_key_iv: "586870987ddf54933fb7cabbf1869d58"
server_key_version: "encryptionkey@0"
I20260927 14:55:52.640920 21975 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6936m5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790520684602251-392-0/minicluster-data/master-0/wal/instance:
uuid: "9a66b153d01345e4a8ff9aa8a117eb76"
format_stamp: "Formatted at 2026-09-27 14:55:52 on dist-test-slave-xk1g"
server_key: "51b68d69baa1760bc14997b2ca7b8dc5"
server_key_iv: "586870987ddf54933fb7cabbf1869d58"
server_key_version: "encryptionkey@0"
I20260927 14:55:52.644466 21975 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260927 14:55:52.646893 21990 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 14:55:52.648089 21975 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s
I20260927 14:55:52.648233 21975 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790520684602251-392-0/minicluster-data/master-0/data,/tmp/dist-test-task6936m5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790520684602251-392-0/minicluster-data/master-0/wal
uuid: "9a66b153d01345e4a8ff9aa8a117eb76"
format_stamp: "Formatted at 2026-09-27 14:55:52 on dist-test-slave-xk1g"
server_key: "51b68d69baa1760bc14997b2ca7b8dc5"
server_key_iv: "586870987ddf54933fb7cabbf1869d58"
server_key_version: "encryptionkey@0"
I20260927 14:55:52.648341 21975 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790520684602251-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790520684602251-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790520684602251-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 14:55:52.665076 21975 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 14:55:52.668223 21975 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 14:55:52.668442 21975 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 14:55:52.675283 21975 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:42493
I20260927 14:55:52.675297 22042 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:42493 every 8 connection(s)
I20260927 14:55:52.676223 21975 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6936m5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790520684602251-392-0/minicluster-data/master-0/data/info.pb
I20260927 14:55:52.678972 22043 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 14:55:52.684187 22043 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9a66b153d01345e4a8ff9aa8a117eb76: Bootstrap starting.
I20260927 14:55:52.685376 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6936m5/build/debug/bin/kudu as pid 21975
I20260927 14:55:52.685530 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6936m5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790520684602251-392-0/minicluster-data/master-0/wal/instance
I20260927 14:55:52.685779 392 external_mini_cluster.cc:1428] Setting key 7b9ca743908b5c21eb63bd98e051a7ef
I20260927 14:55:52.686930 22043 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9a66b153d01345e4a8ff9aa8a117eb76: Neither blocks nor log segments found. Creating new log.
I20260927 14:55:52.687806 22043 log.cc:824] T 00000000000000000000000000000000 P 9a66b153d01345e4a8ff9aa8a117eb76: Log is configured to *not* fsync() on all Append() calls
I20260927 14:55:52.689956 22043 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9a66b153d01345e4a8ff9aa8a117eb76: No bootstrap required, opened a new log
I20260927 14:55:52.693913 22043 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9a66b153d01345e4a8ff9aa8a117eb76 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a66b153d01345e4a8ff9aa8a117eb76" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42493 } }
I20260927 14:55:52.694165 22043 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9a66b153d01345e4a8ff9aa8a117eb76 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 14:55:52.694283 22043 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9a66b153d01345e4a8ff9aa8a117eb76 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a66b153d01345e4a8ff9aa8a117eb76, State: Initialized, Role: FOLLOWER
Sep 27 14:55:52 dist-test-slave-xk1g krb5kdc[21962](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790520952, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 14:55:52.694800 22043 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9a66b153d01345e4a8ff9aa8a117eb76 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a66b153d01345e4a8ff9aa8a117eb76" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42493 } }
I20260927 14:55:52.694985 22043 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9a66b153d01345e4a8ff9aa8a117eb76 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 14:55:52.695060 22043 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9a66b153d01345e4a8ff9aa8a117eb76 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 14:55:52.695170 22043 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9a66b153d01345e4a8ff9aa8a117eb76 [term 0 FOLLOWER]: Advancing to term 1
I20260927 14:55:52.696241 22043 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9a66b153d01345e4a8ff9aa8a117eb76 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a66b153d01345e4a8ff9aa8a117eb76" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42493 } }
I20260927 14:55:52.696615 22043 leader_election.cc:302] T 00000000000000000000000000000000 P 9a66b153d01345e4a8ff9aa8a117eb76 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9a66b153d01345e4a8ff9aa8a117eb76; no voters:
I20260927 14:55:52.696954 22043 leader_election.cc:288] T 00000000000000000000000000000000 P 9a66b153d01345e4a8ff9aa8a117eb76 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 14:55:52.697047 22048 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9a66b153d01345e4a8ff9aa8a117eb76 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 14:55:52.697278 22048 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9a66b153d01345e4a8ff9aa8a117eb76 [term 1 LEADER]: Becoming Leader. State: Replica: 9a66b153d01345e4a8ff9aa8a117eb76, State: Running, Role: LEADER
I20260927 14:55:52.697628 22048 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9a66b153d01345e4a8ff9aa8a117eb76 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a66b153d01345e4a8ff9aa8a117eb76" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42493 } }
I20260927 14:55:52.698227 22043 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9a66b153d01345e4a8ff9aa8a117eb76 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 14:55:52.699208 22049 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9a66b153d01345e4a8ff9aa8a117eb76 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9a66b153d01345e4a8ff9aa8a117eb76" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a66b153d01345e4a8ff9aa8a117eb76" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42493 } } }
I20260927 14:55:52.699412 22049 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9a66b153d01345e4a8ff9aa8a117eb76 [sys.catalog]: This master's current role is: LEADER
I20260927 14:55:52.699674 22049 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9a66b153d01345e4a8ff9aa8a117eb76 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9a66b153d01345e4a8ff9aa8a117eb76. Latest consensus state: current_term: 1 leader_uuid: "9a66b153d01345e4a8ff9aa8a117eb76" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a66b153d01345e4a8ff9aa8a117eb76" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42493 } } }
I20260927 14:55:52.699808 22049 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9a66b153d01345e4a8ff9aa8a117eb76 [sys.catalog]: This master's current role is: LEADER
I20260927 14:55:52.701082 22057 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 14:55:52.701562 22046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:55:52.686810 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48833 (local address 127.0.98.62:42493)
0927 14:55:52.687228 (+ 418us) server_negotiation.cc:206] Beginning negotiation
0927 14:55:52.687248 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0927 14:55:52.687288 (+ 40us) server_negotiation.cc:407] Connection header received
0927 14:55:52.688158 (+ 870us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:55:52.688182 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:55:52.688561 (+ 379us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:55:52.688904 (+ 343us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:55:52.689620 (+ 716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:55:52.691050 (+ 1430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:55:52.691855 (+ 805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:55:52.692624 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:55:52.692821 (+ 197us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:55:52.695533 (+ 2712us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:55:52.695538 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:55:52.695550 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 14:55:52.695575 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:55:52.697770 (+ 2195us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:55:52.698319 (+ 549us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:55:52.698325 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:55:52.698331 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:55:52.698478 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:55:52.698901 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:55:52.698906 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:55:52.698908 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:55:52.699402 (+ 494us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:55:52.700195 (+ 793us) server_negotiation.cc:1035] Waiting for connection context
0927 14:55:52.700373 (+ 178us) server_negotiation.cc:299] Negotiation successful
0927 14:55:52.700584 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":263,"thread_start_us":119,"threads_started":1}
I20260927 14:55:52.703958 22057 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 14:55:52.704762 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 14:55:52.709896 22057 catalog_manager.cc:1347] Generated new cluster ID: d7837e5ce3824d4280c4b9f112767bd8
I20260927 14:55:52.709967 22057 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 14:55:52.715536 22046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:55:52.706925 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46939 (local address 127.0.98.62:42493)
0927 14:55:52.707028 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0927 14:55:52.707032 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 14:55:52.707235 (+ 203us) server_negotiation.cc:407] Connection header received
0927 14:55:52.707372 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:55:52.707374 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:55:52.707414 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:55:52.707503 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:55:52.708253 (+ 750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:55:52.709670 (+ 1417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:55:52.710560 (+ 890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:55:52.710766 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:55:52.710882 (+ 116us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:55:52.712247 (+ 1365us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:55:52.712251 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:55:52.712254 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 14:55:52.712277 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:55:52.713769 (+ 1492us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:55:52.714251 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:55:52.714255 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:55:52.714257 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:55:52.714338 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:55:52.714661 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:55:52.714664 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:55:52.714666 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:55:52.714848 (+ 182us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:55:52.715007 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0927 14:55:52.715274 (+ 267us) server_negotiation.cc:299] Negotiation successful
0927 14:55:52.715369 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260927 14:55:52.719425 22057 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 14:55:52.720669 22057 catalog_manager.cc:1504] Loading token signing keys...
I20260927 14:55:52.728428 22057 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9a66b153d01345e4a8ff9aa8a117eb76: Generated new TSK 0
I20260927 14:55:52.729122 22057 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 14:55:52.736230 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6936m5/build/debug/bin/kudu with pid 21975
W20260927 14:55:52.741448 21960 connection.cc:535] client connection to 127.0.98.62:42493 recv error: Network error: failed to read from TLS socket (remote: 127.0.98.62:42493): Connection reset by peer (error 104)
2026-09-27T14:55:52Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (1388 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-task6936m5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790520684602251-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 14:55:52 dist-test-slave-xk1g krb5kdc[22074](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 14:55:52 dist-test-slave-xk1g krb5kdc[22074](info): set up 2 sockets
Sep 27 14:55:52 dist-test-slave-xk1g krb5kdc[22074](info): commencing operation
krb5kdc: starting...
W20260927 14:55:53.781667 392 mini_kdc.cc:121] Time spent starting KDC: real 1.026s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6936m5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790520684602251-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6936m5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790520684602251-392-0/krb5kdc/test-user.keytab.
Sep 27 14:55:53 dist-test-slave-xk1g krb5kdc[22074](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790520953, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T14:55:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T14:55:53Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6936m5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790520684602251-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6936m5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790520684602251-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 14:55:53.903651 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6936m5/build/debug/bin/kudu
/tmp/dist-test-task6936m5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790520684602251-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790520684602251-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790520684602251-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790520684602251-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40501
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790520684602251-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41377
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:40501
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790520684602251-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790520684602251-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790520684602251-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 14:55:54.010778 22087 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 14:55:54.011049 22087 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 14:55:54.011126 22087 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 14:55:54.015105 22087 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 14:55:54.015161 22087 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 14:55:54.015206 22087 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 14:55:54.015245 22087 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 14:55:54.015280 22087 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 14:55:54.020370 22087 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41377
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790520684602251-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790520684602251-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:40501
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790520684602251-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40501
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790520684602251-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22087
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790520684602251-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:15:17 UTC on ccycloud.quasar-ledxfe.root.comops.site
build id 4676134
I20260927 14:55:54.021379 22087 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 14:55:54.022363 22087 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 14:55:54.030340 22093 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 14:55:54.030391 22095 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 14:55:54.030486 22087 server_base.cc:1034] running on GCE node
W20260927 14:55:54.030391 22092 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 14:55:54.031224 22087 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 14:55:54.032189 22087 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 14:55:54.033341 22087 hybrid_clock.cc:648] HybridClock initialized: now 1790520954033328 us; error 26 us; skew 500 ppm
Sep 27 14:55:54 dist-test-slave-xk1g krb5kdc[22074](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790520954, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 14:55:54.035851 22087 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 14:55:54.036648 22087 webserver.cc:460] Webserver started at http://127.0.98.62:45703/ using document root <none> and password file <none>
I20260927 14:55:54.037235 22087 fs_manager.cc:359] Metadata directory not provided
I20260927 14:55:54.037321 22087 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 14:55:54.037556 22087 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 14:55:54.040019 22087 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6936m5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790520684602251-392-0/minicluster-data/master-0/data/instance:
uuid: "f755a521ce954c26ba6d32c98cd8cdd6"
format_stamp: "Formatted at 2026-09-27 14:55:54 on dist-test-slave-xk1g"
server_key: "8291bb5161661b801bf0d746da4adca8"
server_key_iv: "6ef571281ab598d1b2844b4a9dab16c5"
server_key_version: "encryptionkey@0"
I20260927 14:55:54.040568 22087 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6936m5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790520684602251-392-0/minicluster-data/master-0/wal/instance:
uuid: "f755a521ce954c26ba6d32c98cd8cdd6"
format_stamp: "Formatted at 2026-09-27 14:55:54 on dist-test-slave-xk1g"
server_key: "8291bb5161661b801bf0d746da4adca8"
server_key_iv: "6ef571281ab598d1b2844b4a9dab16c5"
server_key_version: "encryptionkey@0"
I20260927 14:55:54.043990 22087 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.003s
I20260927 14:55:54.046296 22102 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 14:55:54.047454 22087 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 14:55:54.047591 22087 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790520684602251-392-0/minicluster-data/master-0/data,/tmp/dist-test-task6936m5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790520684602251-392-0/minicluster-data/master-0/wal
uuid: "f755a521ce954c26ba6d32c98cd8cdd6"
format_stamp: "Formatted at 2026-09-27 14:55:54 on dist-test-slave-xk1g"
server_key: "8291bb5161661b801bf0d746da4adca8"
server_key_iv: "6ef571281ab598d1b2844b4a9dab16c5"
server_key_version: "encryptionkey@0"
I20260927 14:55:54.047701 22087 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790520684602251-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790520684602251-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790520684602251-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 14:55:54.063318 22087 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 14:55:54.066201 22087 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 14:55:54.066398 22087 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 14:55:54.072887 22087 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:40501
I20260927 14:55:54.072963 22154 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:40501 every 8 connection(s)
I20260927 14:55:54.073849 22087 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6936m5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790520684602251-392-0/minicluster-data/master-0/data/info.pb
I20260927 14:55:54.076463 22155 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 14:55:54.079296 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6936m5/build/debug/bin/kudu as pid 22087
I20260927 14:55:54.079452 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6936m5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790520684602251-392-0/minicluster-data/master-0/wal/instance
I20260927 14:55:54.079702 392 external_mini_cluster.cc:1428] Setting key a8bb917b4b4c31aa31dafd6cf060f682
I20260927 14:55:54.082310 22155 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f755a521ce954c26ba6d32c98cd8cdd6: Bootstrap starting.
I20260927 14:55:54.084929 22155 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f755a521ce954c26ba6d32c98cd8cdd6: Neither blocks nor log segments found. Creating new log.
I20260927 14:55:54.085728 22155 log.cc:824] T 00000000000000000000000000000000 P f755a521ce954c26ba6d32c98cd8cdd6: Log is configured to *not* fsync() on all Append() calls
I20260927 14:55:54.087888 22155 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f755a521ce954c26ba6d32c98cd8cdd6: No bootstrap required, opened a new log
I20260927 14:55:54.089478 22158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:55:54.081113 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49899 (local address 127.0.98.62:40501)
0927 14:55:54.081539 (+ 426us) server_negotiation.cc:206] Beginning negotiation
0927 14:55:54.081558 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0927 14:55:54.081600 (+ 42us) server_negotiation.cc:407] Connection header received
0927 14:55:54.082388 (+ 788us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:55:54.082410 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:55:54.082812 (+ 402us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:55:54.083167 (+ 355us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:55:54.083596 (+ 429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:55:54.085075 (+ 1479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:55:54.085995 (+ 920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:55:54.086595 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:55:54.086740 (+ 145us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:55:54.087550 (+ 810us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:55:54.087556 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:55:54.087572 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 14:55:54.088061 (+ 489us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:55:54.088127 (+ 66us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 14:55:54.088156 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:55:54.088298 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0927 14:55:54.088418 (+ 120us) server_negotiation.cc:299] Negotiation successful
0927 14:55:54.088548 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":275,"thread_start_us":109,"threads_started":1}
I20260927 14:55:54.092348 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 14:55:54.092283 22155 raft_consensus.cc:348] T 00000000000000000000000000000000 P f755a521ce954c26ba6d32c98cd8cdd6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f755a521ce954c26ba6d32c98cd8cdd6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40501 } }
I20260927 14:55:54.092542 22155 raft_consensus.cc:374] T 00000000000000000000000000000000 P f755a521ce954c26ba6d32c98cd8cdd6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 14:55:54.092615 22155 raft_consensus.cc:729] T 00000000000000000000000000000000 P f755a521ce954c26ba6d32c98cd8cdd6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f755a521ce954c26ba6d32c98cd8cdd6, State: Initialized, Role: FOLLOWER
I20260927 14:55:54.093106 22155 consensus_queue.cc:260] T 00000000000000000000000000000000 P f755a521ce954c26ba6d32c98cd8cdd6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f755a521ce954c26ba6d32c98cd8cdd6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40501 } }
I20260927 14:55:54.093241 22155 raft_consensus.cc:388] T 00000000000000000000000000000000 P f755a521ce954c26ba6d32c98cd8cdd6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 14:55:54.093323 22155 raft_consensus.cc:482] T 00000000000000000000000000000000 P f755a521ce954c26ba6d32c98cd8cdd6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 14:55:54.093417 22155 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f755a521ce954c26ba6d32c98cd8cdd6 [term 0 FOLLOWER]: Advancing to term 1
I20260927 14:55:54.094456 22155 raft_consensus.cc:504] T 00000000000000000000000000000000 P f755a521ce954c26ba6d32c98cd8cdd6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f755a521ce954c26ba6d32c98cd8cdd6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40501 } }
I20260927 14:55:54.094823 22155 leader_election.cc:302] T 00000000000000000000000000000000 P f755a521ce954c26ba6d32c98cd8cdd6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f755a521ce954c26ba6d32c98cd8cdd6; no voters:
I20260927 14:55:54.095180 22155 leader_election.cc:288] T 00000000000000000000000000000000 P f755a521ce954c26ba6d32c98cd8cdd6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 14:55:54.095278 22165 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f755a521ce954c26ba6d32c98cd8cdd6 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 14:55:54.095510 22165 raft_consensus.cc:686] T 00000000000000000000000000000000 P f755a521ce954c26ba6d32c98cd8cdd6 [term 1 LEADER]: Becoming Leader. State: Replica: f755a521ce954c26ba6d32c98cd8cdd6, State: Running, Role: LEADER
I20260927 14:55:54.095837 22165 consensus_queue.cc:237] T 00000000000000000000000000000000 P f755a521ce954c26ba6d32c98cd8cdd6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f755a521ce954c26ba6d32c98cd8cdd6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40501 } }
I20260927 14:55:54.096462 22155 sys_catalog.cc:564] T 00000000000000000000000000000000 P f755a521ce954c26ba6d32c98cd8cdd6 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 14:55:54.097368 22167 sys_catalog.cc:455] T 00000000000000000000000000000000 P f755a521ce954c26ba6d32c98cd8cdd6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f755a521ce954c26ba6d32c98cd8cdd6. Latest consensus state: current_term: 1 leader_uuid: "f755a521ce954c26ba6d32c98cd8cdd6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f755a521ce954c26ba6d32c98cd8cdd6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40501 } } }
I20260927 14:55:54.097488 22167 sys_catalog.cc:458] T 00000000000000000000000000000000 P f755a521ce954c26ba6d32c98cd8cdd6 [sys.catalog]: This master's current role is: LEADER
I20260927 14:55:54.097821 22166 sys_catalog.cc:455] T 00000000000000000000000000000000 P f755a521ce954c26ba6d32c98cd8cdd6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f755a521ce954c26ba6d32c98cd8cdd6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f755a521ce954c26ba6d32c98cd8cdd6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40501 } } }
I20260927 14:55:54.097918 22166 sys_catalog.cc:458] T 00000000000000000000000000000000 P f755a521ce954c26ba6d32c98cd8cdd6 [sys.catalog]: This master's current role is: LEADER
I20260927 14:55:54.099653 22158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:55:54.094567 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48827 (local address 127.0.98.62:40501)
0927 14:55:54.094677 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0927 14:55:54.094681 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 14:55:54.094983 (+ 302us) server_negotiation.cc:407] Connection header received
0927 14:55:54.095068 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:55:54.095071 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:55:54.095130 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:55:54.095214 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:55:54.095629 (+ 415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:55:54.096370 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:55:54.097610 (+ 1240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:55:54.097987 (+ 377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:55:54.098076 (+ 89us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:55:54.099201 (+ 1125us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:55:54.099204 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:55:54.099206 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 14:55:54.099217 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:55:54.099240 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 14:55:54.099250 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:55:54.099320 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0927 14:55:54.099471 (+ 151us) server_negotiation.cc:299] Negotiation successful
0927 14:55:54.099498 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
W20260927 14:55:54.101563 22181 catalog_manager.cc:1558] T 00000000000000000000000000000000 P f755a521ce954c26ba6d32c98cd8cdd6: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 14:55:54.101649 22181 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 14:55:54.101724 22174 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 14:55:54.102382 22174 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 14:55:54.107682 22174 catalog_manager.cc:1347] Generated new cluster ID: 745a437968ed4196b4b23a74268f1dfe
I20260927 14:55:54.107750 22174 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 14:55:54.131879 22174 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 14:55:54.132896 22174 catalog_manager.cc:1504] Loading token signing keys...
I20260927 14:55:54.142582 22174 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f755a521ce954c26ba6d32c98cd8cdd6: Generated new TSK 0
I20260927 14:55:54.143323 22174 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 14:55:54.146986 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6936m5/build/debug/bin/kudu with pid 22087
W20260927 14:55:54.152149 22072 connection.cc:535] client connection to 127.0.98.62:40501 recv error: Network error: failed to read from TLS socket (remote: 127.0.98.62:40501): Connection reset by peer (error 104)
2026-09-27T14:55:54Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1410 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-task6936m5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790520684602251-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 14:55:54 dist-test-slave-xk1g krb5kdc[22186](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 14:55:54 dist-test-slave-xk1g krb5kdc[22186](info): set up 2 sockets
Sep 27 14:55:54 dist-test-slave-xk1g krb5kdc[22186](info): commencing operation
krb5kdc: starting...
W20260927 14:55:55.177330 392 mini_kdc.cc:121] Time spent starting KDC: real 1.011s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6936m5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790520684602251-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6936m5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790520684602251-392-0/krb5kdc/test-user.keytab.
Sep 27 14:55:55 dist-test-slave-xk1g krb5kdc[22186](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790520955, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T14:55:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T14:55:55Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6936m5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790520684602251-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6936m5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790520684602251-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 14:55:55.294358 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6936m5/build/debug/bin/kudu
/tmp/dist-test-task6936m5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790520684602251-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790520684602251-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790520684602251-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790520684602251-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40315
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790520684602251-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42947
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:40315
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790520684602251-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790520684602251-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790520684602251-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 14:55:55.404268 22199 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 14:55:55.404547 22199 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 14:55:55.404593 22199 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 14:55:55.408565 22199 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 14:55:55.408627 22199 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 14:55:55.408650 22199 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 14:55:55.408668 22199 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 14:55:55.408686 22199 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 14:55:55.413925 22199 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42947
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790520684602251-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790520684602251-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:40315
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790520684602251-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40315
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790520684602251-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22199
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790520684602251-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:15:17 UTC on ccycloud.quasar-ledxfe.root.comops.site
build id 4676134
I20260927 14:55:55.414911 22199 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 14:55:55.415838 22199 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 14:55:55.424268 22207 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 14:55:55.424325 22199 server_base.cc:1034] running on GCE node
W20260927 14:55:55.424304 22205 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 14:55:55.424268 22204 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 14:55:55.425158 22199 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 14:55:55.426246 22199 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 14:55:55.427454 22199 hybrid_clock.cc:648] HybridClock initialized: now 1790520955427420 us; error 47 us; skew 500 ppm
Sep 27 14:55:55 dist-test-slave-xk1g krb5kdc[22186](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790520955, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 14:55:55.430465 22199 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 14:55:55.431360 22199 webserver.cc:460] Webserver started at http://127.0.98.62:34221/ using document root <none> and password file <none>
I20260927 14:55:55.431926 22199 fs_manager.cc:359] Metadata directory not provided
I20260927 14:55:55.432009 22199 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 14:55:55.432227 22199 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 14:55:55.434260 22199 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6936m5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790520684602251-392-0/minicluster-data/master-0/data/instance:
uuid: "1fafe7cf619d40d8898cd085f212195b"
format_stamp: "Formatted at 2026-09-27 14:55:55 on dist-test-slave-xk1g"
server_key: "547888a5f2092f7aaf0fa1294d4293d9"
server_key_iv: "b7ea57200926b6880e0d5ad6446d4947"
server_key_version: "encryptionkey@0"
I20260927 14:55:55.434815 22199 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6936m5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790520684602251-392-0/minicluster-data/master-0/wal/instance:
uuid: "1fafe7cf619d40d8898cd085f212195b"
format_stamp: "Formatted at 2026-09-27 14:55:55 on dist-test-slave-xk1g"
server_key: "547888a5f2092f7aaf0fa1294d4293d9"
server_key_iv: "b7ea57200926b6880e0d5ad6446d4947"
server_key_version: "encryptionkey@0"
I20260927 14:55:55.438369 22199 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 14:55:55.440696 22214 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 14:55:55.441870 22199 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 14:55:55.442013 22199 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790520684602251-392-0/minicluster-data/master-0/data,/tmp/dist-test-task6936m5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790520684602251-392-0/minicluster-data/master-0/wal
uuid: "1fafe7cf619d40d8898cd085f212195b"
format_stamp: "Formatted at 2026-09-27 14:55:55 on dist-test-slave-xk1g"
server_key: "547888a5f2092f7aaf0fa1294d4293d9"
server_key_iv: "b7ea57200926b6880e0d5ad6446d4947"
server_key_version: "encryptionkey@0"
I20260927 14:55:55.442112 22199 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790520684602251-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790520684602251-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790520684602251-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 14:55:55.446475 22199 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 14:55:55.449225 22199 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 14:55:55.449402 22199 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 14:55:55.455987 22199 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:40315
I20260927 14:55:55.456008 22266 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:40315 every 8 connection(s)
I20260927 14:55:55.456789 22199 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6936m5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790520684602251-392-0/minicluster-data/master-0/data/info.pb
I20260927 14:55:55.459319 22267 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 14:55:55.459903 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6936m5/build/debug/bin/kudu as pid 22199
I20260927 14:55:55.460041 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6936m5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790520684602251-392-0/minicluster-data/master-0/wal/instance
I20260927 14:55:55.460306 392 external_mini_cluster.cc:1428] Setting key 7e52a28fd823055085258b036768b9f3
I20260927 14:55:55.465606 22270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:55:55.461644 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42245 (local address 127.0.98.62:40315)
0927 14:55:55.462050 (+ 406us) server_negotiation.cc:206] Beginning negotiation
0927 14:55:55.462068 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0927 14:55:55.462108 (+ 40us) server_negotiation.cc:407] Connection header received
0927 14:55:55.462923 (+ 815us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:55:55.462946 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:55:55.463292 (+ 346us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:55:55.463629 (+ 337us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:55:55.464258 (+ 629us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:55:55.464263 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:55:55.464277 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 14:55:55.464287 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:55:55.464337 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 14:55:55.464361 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:55:55.464515 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0927 14:55:55.464582 (+ 67us) server_negotiation.cc:299] Negotiation successful
0927 14:55:55.464727 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":250,"thread_start_us":98,"threads_started":1}
I20260927 14:55:55.465955 22267 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1fafe7cf619d40d8898cd085f212195b: Bootstrap starting.
I20260927 14:55:55.468421 22267 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1fafe7cf619d40d8898cd085f212195b: Neither blocks nor log segments found. Creating new log.
I20260927 14:55:55.468504 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 14:55:55.469347 22267 log.cc:824] T 00000000000000000000000000000000 P 1fafe7cf619d40d8898cd085f212195b: Log is configured to *not* fsync() on all Append() calls
I20260927 14:55:55.471552 22267 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1fafe7cf619d40d8898cd085f212195b: No bootstrap required, opened a new log
I20260927 14:55:55.472354 22270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:55:55.470735 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47869 (local address 127.0.98.62:40315)
0927 14:55:55.470849 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0927 14:55:55.470854 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 14:55:55.471004 (+ 150us) server_negotiation.cc:407] Connection header received
0927 14:55:55.471130 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:55:55.471132 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:55:55.471159 (+ 27us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:55:55.471215 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:55:55.471966 (+ 751us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:55:55.471971 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:55:55.471974 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 14:55:55.471985 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:55:55.472009 (+ 24us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 14:55:55.472027 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:55:55.472115 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0927 14:55:55.472204 (+ 89us) server_negotiation.cc:299] Negotiation successful
0927 14:55:55.472240 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260927 14:55:55.475191 22267 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1fafe7cf619d40d8898cd085f212195b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1fafe7cf619d40d8898cd085f212195b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40315 } }
I20260927 14:55:55.475399 22267 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1fafe7cf619d40d8898cd085f212195b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 14:55:55.475432 22267 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1fafe7cf619d40d8898cd085f212195b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1fafe7cf619d40d8898cd085f212195b, State: Initialized, Role: FOLLOWER
I20260927 14:55:55.475781 22267 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1fafe7cf619d40d8898cd085f212195b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1fafe7cf619d40d8898cd085f212195b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40315 } }
I20260927 14:55:55.475888 22267 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1fafe7cf619d40d8898cd085f212195b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 14:55:55.475930 22267 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1fafe7cf619d40d8898cd085f212195b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 14:55:55.475996 22267 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1fafe7cf619d40d8898cd085f212195b [term 0 FOLLOWER]: Advancing to term 1
I20260927 14:55:55.476807 22267 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1fafe7cf619d40d8898cd085f212195b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1fafe7cf619d40d8898cd085f212195b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40315 } }
I20260927 14:55:55.477154 22267 leader_election.cc:302] T 00000000000000000000000000000000 P 1fafe7cf619d40d8898cd085f212195b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1fafe7cf619d40d8898cd085f212195b; no voters:
I20260927 14:55:55.477409 22267 leader_election.cc:288] T 00000000000000000000000000000000 P 1fafe7cf619d40d8898cd085f212195b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 14:55:55.477524 22277 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1fafe7cf619d40d8898cd085f212195b [term 1 FOLLOWER]: Leader election won for term 1
I20260927 14:55:55.477756 22277 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1fafe7cf619d40d8898cd085f212195b [term 1 LEADER]: Becoming Leader. State: Replica: 1fafe7cf619d40d8898cd085f212195b, State: Running, Role: LEADER
I20260927 14:55:55.478088 22277 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1fafe7cf619d40d8898cd085f212195b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1fafe7cf619d40d8898cd085f212195b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40315 } }
I20260927 14:55:55.478631 22267 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1fafe7cf619d40d8898cd085f212195b [sys.catalog]: configured and running, proceeding with master startup.
I20260927 14:55:55.479737 22278 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1fafe7cf619d40d8898cd085f212195b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1fafe7cf619d40d8898cd085f212195b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1fafe7cf619d40d8898cd085f212195b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40315 } } }
I20260927 14:55:55.479748 22279 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1fafe7cf619d40d8898cd085f212195b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1fafe7cf619d40d8898cd085f212195b. Latest consensus state: current_term: 1 leader_uuid: "1fafe7cf619d40d8898cd085f212195b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1fafe7cf619d40d8898cd085f212195b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40315 } } }
I20260927 14:55:55.479882 22278 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1fafe7cf619d40d8898cd085f212195b [sys.catalog]: This master's current role is: LEADER
I20260927 14:55:55.479887 22279 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1fafe7cf619d40d8898cd085f212195b [sys.catalog]: This master's current role is: LEADER
I20260927 14:55:55.480216 22286 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 14:55:55.482805 22286 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 14:55:55.487673 22286 catalog_manager.cc:1347] Generated new cluster ID: 34dab39f1a3c4ec788dbbc87942ff50e
I20260927 14:55:55.487737 22286 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 14:55:55.541531 22286 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 14:55:55.541918 22286 catalog_manager.cc:1504] Loading token signing keys...
I20260927 14:55:55.550698 22286 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1fafe7cf619d40d8898cd085f212195b: Generated new TSK 0
I20260927 14:55:55.551376 22286 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 14:55:55.562803 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6936m5/build/debug/bin/kudu with pid 22199
2026-09-27T14:55:55Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1416 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-task6936m5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790520684602251-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 14:55:55 dist-test-slave-xk1g krb5kdc[22298](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 14:55:55 dist-test-slave-xk1g krb5kdc[22298](info): set up 2 sockets
Sep 27 14:55:55 dist-test-slave-xk1g krb5kdc[22298](info): commencing operation
krb5kdc: starting...
W20260927 14:55:56.597344 392 mini_kdc.cc:121] Time spent starting KDC: real 1.015s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6936m5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790520684602251-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6936m5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790520684602251-392-0/krb5kdc/test-user.keytab.
Sep 27 14:55:56 dist-test-slave-xk1g krb5kdc[22298](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790520956, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T14:55:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T14:55:56Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6936m5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790520684602251-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6936m5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790520684602251-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 14:55:56.716094 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6936m5/build/debug/bin/kudu
/tmp/dist-test-task6936m5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790520684602251-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790520684602251-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790520684602251-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790520684602251-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46693
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790520684602251-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35517
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:46693
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790520684602251-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790520684602251-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790520684602251-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 14:55:56.821794 22311 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 14:55:56.822021 22311 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 14:55:56.822058 22311 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 14:55:56.825886 22311 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 14:55:56.825932 22311 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 14:55:56.825950 22311 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 14:55:56.825965 22311 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 14:55:56.825979 22311 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 14:55:56.830770 22311 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35517
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790520684602251-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790520684602251-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:46693
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790520684602251-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46693
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790520684602251-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22311
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790520684602251-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:15:17 UTC on ccycloud.quasar-ledxfe.root.comops.site
build id 4676134
I20260927 14:55:56.831693 22311 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 14:55:56.832510 22311 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 14:55:56.840721 22319 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 14:55:56.840799 22317 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 14:55:56.840840 22316 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 14:55:56.840889 22311 server_base.cc:1034] running on GCE node
I20260927 14:55:56.841627 22311 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 14:55:56.842666 22311 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 14:55:56.843822 22311 hybrid_clock.cc:648] HybridClock initialized: now 1790520956843810 us; error 25 us; skew 500 ppm
Sep 27 14:55:56 dist-test-slave-xk1g krb5kdc[22298](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790520956, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 14:55:56.850627 22311 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 14:55:56.851554 22311 webserver.cc:460] Webserver started at http://127.0.98.62:46233/ using document root <none> and password file <none>
I20260927 14:55:56.852092 22311 fs_manager.cc:359] Metadata directory not provided
I20260927 14:55:56.852150 22311 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 14:55:56.852387 22311 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 14:55:56.854401 22311 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6936m5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790520684602251-392-0/minicluster-data/master-0/data/instance:
uuid: "1b599354886341028c4d0459c9a6c317"
format_stamp: "Formatted at 2026-09-27 14:55:56 on dist-test-slave-xk1g"
server_key: "5dd4d95b062094237ca0c47776314d46"
server_key_iv: "59f6e029c3b03b9755bb1e5699887d4b"
server_key_version: "encryptionkey@0"
I20260927 14:55:56.854943 22311 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6936m5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790520684602251-392-0/minicluster-data/master-0/wal/instance:
uuid: "1b599354886341028c4d0459c9a6c317"
format_stamp: "Formatted at 2026-09-27 14:55:56 on dist-test-slave-xk1g"
server_key: "5dd4d95b062094237ca0c47776314d46"
server_key_iv: "59f6e029c3b03b9755bb1e5699887d4b"
server_key_version: "encryptionkey@0"
I20260927 14:55:56.858500 22311 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260927 14:55:56.860730 22326 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 14:55:56.862010 22311 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 14:55:56.862143 22311 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790520684602251-392-0/minicluster-data/master-0/data,/tmp/dist-test-task6936m5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790520684602251-392-0/minicluster-data/master-0/wal
uuid: "1b599354886341028c4d0459c9a6c317"
format_stamp: "Formatted at 2026-09-27 14:55:56 on dist-test-slave-xk1g"
server_key: "5dd4d95b062094237ca0c47776314d46"
server_key_iv: "59f6e029c3b03b9755bb1e5699887d4b"
server_key_version: "encryptionkey@0"
I20260927 14:55:56.862250 22311 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790520684602251-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790520684602251-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790520684602251-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 14:55:56.877192 22311 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 14:55:56.879688 22311 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 14:55:56.879879 22311 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 14:55:56.886803 22311 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:46693
I20260927 14:55:56.886829 22378 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:46693 every 8 connection(s)
I20260927 14:55:56.887673 22311 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6936m5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790520684602251-392-0/minicluster-data/master-0/data/info.pb
I20260927 14:55:56.890177 22379 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 14:55:56.891964 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6936m5/build/debug/bin/kudu as pid 22311
I20260927 14:55:56.892102 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6936m5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790520684602251-392-0/minicluster-data/master-0/wal/instance
I20260927 14:55:56.892362 392 external_mini_cluster.cc:1428] Setting key 77fef3712c0abe09568aee5d5c1b676c
I20260927 14:55:56.896651 22379 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1b599354886341028c4d0459c9a6c317: Bootstrap starting.
I20260927 14:55:56.899127 22379 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1b599354886341028c4d0459c9a6c317: Neither blocks nor log segments found. Creating new log.
I20260927 14:55:56.899871 22379 log.cc:824] T 00000000000000000000000000000000 P 1b599354886341028c4d0459c9a6c317: Log is configured to *not* fsync() on all Append() calls
Sep 27 14:55:56 dist-test-slave-xk1g krb5kdc[22298](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790520956, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 14:55:56.902427 22379 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1b599354886341028c4d0459c9a6c317: No bootstrap required, opened a new log
I20260927 14:55:56.906352 22379 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1b599354886341028c4d0459c9a6c317 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b599354886341028c4d0459c9a6c317" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46693 } }
I20260927 14:55:56.906569 22379 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1b599354886341028c4d0459c9a6c317 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 14:55:56.906601 22379 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1b599354886341028c4d0459c9a6c317 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b599354886341028c4d0459c9a6c317, State: Initialized, Role: FOLLOWER
I20260927 14:55:56.907054 22379 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1b599354886341028c4d0459c9a6c317 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b599354886341028c4d0459c9a6c317" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46693 } }
I20260927 14:55:56.907114 22382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:55:56.893667 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42329 (local address 127.0.98.62:46693)
0927 14:55:56.894063 (+ 396us) server_negotiation.cc:206] Beginning negotiation
0927 14:55:56.894081 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0927 14:55:56.894125 (+ 44us) server_negotiation.cc:407] Connection header received
0927 14:55:56.894855 (+ 730us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:55:56.894876 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:55:56.895275 (+ 399us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:55:56.895624 (+ 349us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:55:56.896254 (+ 630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:55:56.897684 (+ 1430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:55:56.898506 (+ 822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:55:56.899161 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:55:56.899327 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:55:56.902263 (+ 2936us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:55:56.902268 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:55:56.902280 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 14:55:56.902306 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:55:56.904017 (+ 1711us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:55:56.904564 (+ 547us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:55:56.904568 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:55:56.904573 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:55:56.904677 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:55:56.905018 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:55:56.905020 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:55:56.905021 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:55:56.905479 (+ 458us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:55:56.905640 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0927 14:55:56.905929 (+ 289us) server_negotiation.cc:299] Negotiation successful
0927 14:55:56.906257 (+ 328us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":249,"thread_start_us":89,"threads_started":1}
I20260927 14:55:56.907168 22379 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1b599354886341028c4d0459c9a6c317 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 14:55:56.907243 22379 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1b599354886341028c4d0459c9a6c317 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 14:55:56.907332 22379 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1b599354886341028c4d0459c9a6c317 [term 0 FOLLOWER]: Advancing to term 1
I20260927 14:55:56.908201 22379 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1b599354886341028c4d0459c9a6c317 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b599354886341028c4d0459c9a6c317" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46693 } }
I20260927 14:55:56.908536 22379 leader_election.cc:302] T 00000000000000000000000000000000 P 1b599354886341028c4d0459c9a6c317 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1b599354886341028c4d0459c9a6c317; no voters:
I20260927 14:55:56.908859 22379 leader_election.cc:288] T 00000000000000000000000000000000 P 1b599354886341028c4d0459c9a6c317 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 14:55:56.908928 22384 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1b599354886341028c4d0459c9a6c317 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 14:55:56.909188 22384 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1b599354886341028c4d0459c9a6c317 [term 1 LEADER]: Becoming Leader. State: Replica: 1b599354886341028c4d0459c9a6c317, State: Running, Role: LEADER
I20260927 14:55:56.909523 22384 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1b599354886341028c4d0459c9a6c317 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b599354886341028c4d0459c9a6c317" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46693 } }
I20260927 14:55:56.909888 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 14:55:56.910169 22379 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1b599354886341028c4d0459c9a6c317 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 14:55:56.911213 22385 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1b599354886341028c4d0459c9a6c317 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1b599354886341028c4d0459c9a6c317" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b599354886341028c4d0459c9a6c317" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46693 } } }
I20260927 14:55:56.911338 22385 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1b599354886341028c4d0459c9a6c317 [sys.catalog]: This master's current role is: LEADER
I20260927 14:55:56.911684 22386 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1b599354886341028c4d0459c9a6c317 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1b599354886341028c4d0459c9a6c317. Latest consensus state: current_term: 1 leader_uuid: "1b599354886341028c4d0459c9a6c317" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b599354886341028c4d0459c9a6c317" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46693 } } }
I20260927 14:55:56.911794 22386 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1b599354886341028c4d0459c9a6c317 [sys.catalog]: This master's current role is: LEADER
I20260927 14:55:56.912556 22397 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 14:55:56.915875 22397 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 14:55:56.921375 22397 catalog_manager.cc:1347] Generated new cluster ID: 12b2a612f4df4513875e996c28ccc545
I20260927 14:55:56.921489 22397 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 14:55:56.922034 22382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:55:56.912776 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33191 (local address 127.0.98.62:46693)
0927 14:55:56.912892 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0927 14:55:56.912896 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 14:55:56.913118 (+ 222us) server_negotiation.cc:407] Connection header received
0927 14:55:56.913301 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:55:56.913304 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:55:56.913368 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:55:56.913454 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:55:56.914320 (+ 866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:55:56.915183 (+ 863us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:55:56.916320 (+ 1137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:55:56.916647 (+ 327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:55:56.916745 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:55:56.918500 (+ 1755us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:55:56.918504 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:55:56.918506 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 14:55:56.918531 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:55:56.920173 (+ 1642us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:55:56.920656 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:55:56.920660 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:55:56.920663 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:55:56.920752 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:55:56.921093 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:55:56.921096 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:55:56.921097 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:55:56.921304 (+ 207us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:55:56.921420 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0927 14:55:56.921751 (+ 331us) server_negotiation.cc:299] Negotiation successful
0927 14:55:56.921867 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260927 14:55:56.934887 22397 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 14:55:56.935740 22397 catalog_manager.cc:1504] Loading token signing keys...
I20260927 14:55:56.942019 22397 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1b599354886341028c4d0459c9a6c317: Generated new TSK 0
I20260927 14:55:56.942668 22397 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 14:55:56.949162 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6936m5/build/debug/bin/kudu with pid 22311
2026-09-27T14:55:56Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (1386 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-task6936m5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790520684602251-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 14:55:57 dist-test-slave-xk1g krb5kdc[22410](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 14:55:57 dist-test-slave-xk1g krb5kdc[22410](info): set up 2 sockets
Sep 27 14:55:57 dist-test-slave-xk1g krb5kdc[22410](info): commencing operation
krb5kdc: starting...
W20260927 14:55:57.981369 392 mini_kdc.cc:121] Time spent starting KDC: real 1.012s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6936m5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790520684602251-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6936m5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790520684602251-392-0/krb5kdc/test-user.keytab.
Sep 27 14:55:58 dist-test-slave-xk1g krb5kdc[22410](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790520958, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T14:55:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T14:55:58Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6936m5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790520684602251-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6936m5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790520684602251-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 14:55:58.100217 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6936m5/build/debug/bin/kudu
/tmp/dist-test-task6936m5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790520684602251-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790520684602251-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790520684602251-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790520684602251-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40111
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790520684602251-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35151
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:40111
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790520684602251-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790520684602251-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790520684602251-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 14:55:58.206425 22423 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 14:55:58.206723 22423 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 14:55:58.206806 22423 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 14:55:58.210742 22423 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 14:55:58.210827 22423 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 14:55:58.210865 22423 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 14:55:58.210937 22423 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 14:55:58.210990 22423 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 14:55:58.216095 22423 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35151
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790520684602251-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790520684602251-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:40111
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790520684602251-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40111
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790520684602251-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22423
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790520684602251-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:15:17 UTC on ccycloud.quasar-ledxfe.root.comops.site
build id 4676134
I20260927 14:55:58.217147 22423 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 14:55:58.218171 22423 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 14:55:58.226511 22428 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 14:55:58.226615 22423 server_base.cc:1034] running on GCE node
W20260927 14:55:58.226498 22429 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 14:55:58.226774 22431 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 14:55:58.227371 22423 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 14:55:58.228459 22423 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 14:55:58.229646 22423 hybrid_clock.cc:648] HybridClock initialized: now 1790520958229629 us; error 29 us; skew 500 ppm
Sep 27 14:55:58 dist-test-slave-xk1g krb5kdc[22410](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790520958, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 14:55:58.232365 22423 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 14:55:58.233260 22423 webserver.cc:460] Webserver started at http://127.0.98.62:43153/ using document root <none> and password file <none>
I20260927 14:55:58.233799 22423 fs_manager.cc:359] Metadata directory not provided
I20260927 14:55:58.233889 22423 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 14:55:58.234117 22423 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 14:55:58.236164 22423 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6936m5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790520684602251-392-0/minicluster-data/master-0/data/instance:
uuid: "8ea2401c2b2c4042a71c30c198ed8e67"
format_stamp: "Formatted at 2026-09-27 14:55:58 on dist-test-slave-xk1g"
server_key: "447477332d7fd4858af42d2085069d29"
server_key_iv: "6cc714353a00faa77e759ec853fd3d17"
server_key_version: "encryptionkey@0"
I20260927 14:55:58.236735 22423 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6936m5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790520684602251-392-0/minicluster-data/master-0/wal/instance:
uuid: "8ea2401c2b2c4042a71c30c198ed8e67"
format_stamp: "Formatted at 2026-09-27 14:55:58 on dist-test-slave-xk1g"
server_key: "447477332d7fd4858af42d2085069d29"
server_key_iv: "6cc714353a00faa77e759ec853fd3d17"
server_key_version: "encryptionkey@0"
I20260927 14:55:58.240696 22423 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260927 14:55:58.243149 22438 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 14:55:58.244360 22423 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 14:55:58.244530 22423 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790520684602251-392-0/minicluster-data/master-0/data,/tmp/dist-test-task6936m5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790520684602251-392-0/minicluster-data/master-0/wal
uuid: "8ea2401c2b2c4042a71c30c198ed8e67"
format_stamp: "Formatted at 2026-09-27 14:55:58 on dist-test-slave-xk1g"
server_key: "447477332d7fd4858af42d2085069d29"
server_key_iv: "6cc714353a00faa77e759ec853fd3d17"
server_key_version: "encryptionkey@0"
I20260927 14:55:58.244652 22423 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790520684602251-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790520684602251-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790520684602251-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 14:55:58.284489 22423 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 14:55:58.287551 22423 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 14:55:58.287803 22423 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 14:55:58.295305 22423 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:40111
I20260927 14:55:58.295336 22490 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:40111 every 8 connection(s)
I20260927 14:55:58.296314 22423 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6936m5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790520684602251-392-0/minicluster-data/master-0/data/info.pb
I20260927 14:55:58.296561 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6936m5/build/debug/bin/kudu as pid 22423
I20260927 14:55:58.296705 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6936m5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790520684602251-392-0/minicluster-data/master-0/wal/instance
I20260927 14:55:58.297101 392 external_mini_cluster.cc:1428] Setting key 6e5e5d190755feafa0de070aaf2cb703
I20260927 14:55:58.299365 22491 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 14:55:58.306011 22491 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8ea2401c2b2c4042a71c30c198ed8e67: Bootstrap starting.
I20260927 14:55:58.306244 22494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:55:58.298488 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44303 (local address 127.0.98.62:40111)
0927 14:55:58.298893 (+ 405us) server_negotiation.cc:206] Beginning negotiation
0927 14:55:58.298916 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0927 14:55:58.298950 (+ 34us) server_negotiation.cc:407] Connection header received
0927 14:55:58.299624 (+ 674us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:55:58.299641 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:55:58.299976 (+ 335us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:55:58.300246 (+ 270us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:55:58.300665 (+ 419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:55:58.301728 (+ 1063us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:55:58.303003 (+ 1275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:55:58.303454 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:55:58.303566 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:55:58.304525 (+ 959us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:55:58.304530 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:55:58.304544 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 14:55:58.304917 (+ 373us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:55:58.304972 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 14:55:58.304993 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:55:58.305131 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0927 14:55:58.305268 (+ 137us) server_negotiation.cc:299] Negotiation successful
0927 14:55:58.305501 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":268,"thread_start_us":136,"threads_started":1}
I20260927 14:55:58.308599 22491 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8ea2401c2b2c4042a71c30c198ed8e67: Neither blocks nor log segments found. Creating new log.
I20260927 14:55:58.309199 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 14:55:58.309494 22491 log.cc:824] T 00000000000000000000000000000000 P 8ea2401c2b2c4042a71c30c198ed8e67: Log is configured to *not* fsync() on all Append() calls
I20260927 14:55:58.311822 22491 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8ea2401c2b2c4042a71c30c198ed8e67: No bootstrap required, opened a new log
I20260927 14:55:58.315653 22494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:55:58.311534 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51419 (local address 127.0.98.62:40111)
0927 14:55:58.311656 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0927 14:55:58.311659 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 14:55:58.311818 (+ 159us) server_negotiation.cc:407] Connection header received
0927 14:55:58.312014 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:55:58.312016 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:55:58.312055 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:55:58.312120 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:55:58.312452 (+ 332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:55:58.312937 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:55:58.314077 (+ 1140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:55:58.314286 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:55:58.314362 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:55:58.315228 (+ 866us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:55:58.315231 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:55:58.315233 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 14:55:58.315245 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:55:58.315267 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 14:55:58.315277 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:55:58.315332 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0927 14:55:58.315464 (+ 132us) server_negotiation.cc:299] Negotiation successful
0927 14:55:58.315509 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260927 14:55:58.316581 22491 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8ea2401c2b2c4042a71c30c198ed8e67 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ea2401c2b2c4042a71c30c198ed8e67" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40111 } }
I20260927 14:55:58.316838 22491 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8ea2401c2b2c4042a71c30c198ed8e67 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 14:55:58.316905 22491 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8ea2401c2b2c4042a71c30c198ed8e67 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ea2401c2b2c4042a71c30c198ed8e67, State: Initialized, Role: FOLLOWER
I20260927 14:55:58.317389 22491 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8ea2401c2b2c4042a71c30c198ed8e67 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ea2401c2b2c4042a71c30c198ed8e67" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40111 } }
I20260927 14:55:58.317548 22491 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8ea2401c2b2c4042a71c30c198ed8e67 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 14:55:58.317619 22491 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8ea2401c2b2c4042a71c30c198ed8e67 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 14:55:58.317723 22491 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8ea2401c2b2c4042a71c30c198ed8e67 [term 0 FOLLOWER]: Advancing to term 1
I20260927 14:55:58.318682 22491 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8ea2401c2b2c4042a71c30c198ed8e67 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ea2401c2b2c4042a71c30c198ed8e67" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40111 } }
I20260927 14:55:58.319048 22491 leader_election.cc:302] T 00000000000000000000000000000000 P 8ea2401c2b2c4042a71c30c198ed8e67 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8ea2401c2b2c4042a71c30c198ed8e67; no voters:
I20260927 14:55:58.319495 22491 leader_election.cc:288] T 00000000000000000000000000000000 P 8ea2401c2b2c4042a71c30c198ed8e67 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 14:55:58.319579 22501 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8ea2401c2b2c4042a71c30c198ed8e67 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 14:55:58.319834 22501 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8ea2401c2b2c4042a71c30c198ed8e67 [term 1 LEADER]: Becoming Leader. State: Replica: 8ea2401c2b2c4042a71c30c198ed8e67, State: Running, Role: LEADER
I20260927 14:55:58.320176 22501 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8ea2401c2b2c4042a71c30c198ed8e67 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ea2401c2b2c4042a71c30c198ed8e67" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40111 } }
I20260927 14:55:58.320917 22491 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8ea2401c2b2c4042a71c30c198ed8e67 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 14:55:58.321877 22503 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8ea2401c2b2c4042a71c30c198ed8e67 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8ea2401c2b2c4042a71c30c198ed8e67. Latest consensus state: current_term: 1 leader_uuid: "8ea2401c2b2c4042a71c30c198ed8e67" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ea2401c2b2c4042a71c30c198ed8e67" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40111 } } }
I20260927 14:55:58.322006 22503 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8ea2401c2b2c4042a71c30c198ed8e67 [sys.catalog]: This master's current role is: LEADER
I20260927 14:55:58.322361 22510 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 14:55:58.322328 22502 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8ea2401c2b2c4042a71c30c198ed8e67 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8ea2401c2b2c4042a71c30c198ed8e67" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ea2401c2b2c4042a71c30c198ed8e67" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40111 } } }
I20260927 14:55:58.322994 22502 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8ea2401c2b2c4042a71c30c198ed8e67 [sys.catalog]: This master's current role is: LEADER
I20260927 14:55:58.325371 22510 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 14:55:58.330960 22510 catalog_manager.cc:1347] Generated new cluster ID: e1226d4e9d5945ee85cbffacb02795cd
I20260927 14:55:58.331036 22510 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 14:55:58.336351 22510 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 14:55:58.337584 22510 catalog_manager.cc:1504] Loading token signing keys...
I20260927 14:55:58.345382 22510 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8ea2401c2b2c4042a71c30c198ed8e67: Generated new TSK 0
I20260927 14:55:58.346196 22510 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 14:55:58.352365 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6936m5/build/debug/bin/kudu with pid 22423
2026-09-27T14:55:58Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1402 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-task6936m5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790520684602251-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 14:55:58 dist-test-slave-xk1g krb5kdc[22522](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 14:55:58 dist-test-slave-xk1g krb5kdc[22522](info): set up 2 sockets
Sep 27 14:55:58 dist-test-slave-xk1g krb5kdc[22522](info): commencing operation
krb5kdc: starting...
W20260927 14:55:59.401459 392 mini_kdc.cc:121] Time spent starting KDC: real 1.030s user 0.006s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6936m5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790520684602251-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6936m5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790520684602251-392-0/krb5kdc/test-user.keytab.
Sep 27 14:55:59 dist-test-slave-xk1g krb5kdc[22522](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790520959, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T14:55:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T14:55:59Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6936m5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790520684602251-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6936m5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790520684602251-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 14:55:59.520292 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6936m5/build/debug/bin/kudu
/tmp/dist-test-task6936m5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790520684602251-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790520684602251-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790520684602251-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790520684602251-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38915
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790520684602251-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45471
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:38915
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790520684602251-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790520684602251-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790520684602251-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 14:55:59.628067 22535 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 14:55:59.628331 22535 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 14:55:59.628400 22535 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 14:55:59.632360 22535 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 14:55:59.632444 22535 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 14:55:59.632483 22535 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 14:55:59.632537 22535 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 14:55:59.632575 22535 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 14:55:59.637712 22535 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45471
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790520684602251-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790520684602251-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:38915
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790520684602251-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38915
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790520684602251-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22535
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790520684602251-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:15:17 UTC on ccycloud.quasar-ledxfe.root.comops.site
build id 4676134
I20260927 14:55:59.638695 22535 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 14:55:59.639699 22535 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 14:55:59.648749 22541 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 14:55:59.648784 22543 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 14:55:59.648859 22535 server_base.cc:1034] running on GCE node
W20260927 14:55:59.648785 22540 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 14:55:59.649669 22535 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 14:55:59.650745 22535 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 14:55:59.651968 22535 hybrid_clock.cc:648] HybridClock initialized: now 1790520959651953 us; error 27 us; skew 500 ppm
Sep 27 14:55:59 dist-test-slave-xk1g krb5kdc[22522](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790520959, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 14:55:59.654635 22535 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 14:55:59.655493 22535 webserver.cc:460] Webserver started at http://127.0.98.62:39641/ using document root <none> and password file <none>
I20260927 14:55:59.656038 22535 fs_manager.cc:359] Metadata directory not provided
I20260927 14:55:59.656126 22535 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 14:55:59.656348 22535 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 14:55:59.658351 22535 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6936m5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790520684602251-392-0/minicluster-data/master-0/data/instance:
uuid: "1de6423cda9b43efb51af5002dbbf942"
format_stamp: "Formatted at 2026-09-27 14:55:59 on dist-test-slave-xk1g"
server_key: "37ef53205ca0aac113b6f258a1248f65"
server_key_iv: "0bf2f787c8eaa5a1708a4d34fda781bd"
server_key_version: "encryptionkey@0"
I20260927 14:55:59.658896 22535 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6936m5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790520684602251-392-0/minicluster-data/master-0/wal/instance:
uuid: "1de6423cda9b43efb51af5002dbbf942"
format_stamp: "Formatted at 2026-09-27 14:55:59 on dist-test-slave-xk1g"
server_key: "37ef53205ca0aac113b6f258a1248f65"
server_key_iv: "0bf2f787c8eaa5a1708a4d34fda781bd"
server_key_version: "encryptionkey@0"
I20260927 14:55:59.662386 22535 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 14:55:59.664683 22550 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 14:55:59.665822 22535 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 14:55:59.665957 22535 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790520684602251-392-0/minicluster-data/master-0/data,/tmp/dist-test-task6936m5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790520684602251-392-0/minicluster-data/master-0/wal
uuid: "1de6423cda9b43efb51af5002dbbf942"
format_stamp: "Formatted at 2026-09-27 14:55:59 on dist-test-slave-xk1g"
server_key: "37ef53205ca0aac113b6f258a1248f65"
server_key_iv: "0bf2f787c8eaa5a1708a4d34fda781bd"
server_key_version: "encryptionkey@0"
I20260927 14:55:59.666070 22535 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790520684602251-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790520684602251-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790520684602251-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 14:55:59.670400 22535 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 14:55:59.673210 22535 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 14:55:59.673385 22535 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 14:55:59.680346 22535 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:38915
I20260927 14:55:59.680369 22602 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:38915 every 8 connection(s)
I20260927 14:55:59.681289 22535 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6936m5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790520684602251-392-0/minicluster-data/master-0/data/info.pb
I20260927 14:55:59.683844 22603 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 14:55:59.686061 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6936m5/build/debug/bin/kudu as pid 22535
I20260927 14:55:59.686195 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6936m5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790520684602251-392-0/minicluster-data/master-0/wal/instance
I20260927 14:55:59.686491 392 external_mini_cluster.cc:1428] Setting key 1dc5790a768a80eb399cd8728b0ea54f
I20260927 14:55:59.689774 22603 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1de6423cda9b43efb51af5002dbbf942: Bootstrap starting.
I20260927 14:55:59.691452 22606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:55:59.687801 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38893 (local address 127.0.98.62:38915)
0927 14:55:59.688235 (+ 434us) server_negotiation.cc:206] Beginning negotiation
0927 14:55:59.688249 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 14:55:59.688281 (+ 32us) server_negotiation.cc:407] Connection header received
0927 14:55:59.688895 (+ 614us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:55:59.688910 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:55:59.689253 (+ 343us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:55:59.689579 (+ 326us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:55:59.690324 (+ 745us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:55:59.690329 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:55:59.690350 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 14:55:59.690361 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:55:59.690412 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 14:55:59.690429 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:55:59.690588 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0927 14:55:59.690656 (+ 68us) server_negotiation.cc:299] Negotiation successful
0927 14:55:59.690812 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":293,"thread_start_us":146,"threads_started":1}
I20260927 14:55:59.692062 22603 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1de6423cda9b43efb51af5002dbbf942: Neither blocks nor log segments found. Creating new log.
I20260927 14:55:59.692905 22603 log.cc:824] T 00000000000000000000000000000000 P 1de6423cda9b43efb51af5002dbbf942: Log is configured to *not* fsync() on all Append() calls
I20260927 14:55:59.694216 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 14:55:59.694929 22603 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1de6423cda9b43efb51af5002dbbf942: No bootstrap required, opened a new log
I20260927 14:55:59.697800 22606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:55:59.696186 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35283 (local address 127.0.98.62:38915)
0927 14:55:59.696307 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0927 14:55:59.696310 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 14:55:59.696434 (+ 124us) server_negotiation.cc:407] Connection header received
0927 14:55:59.696554 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:55:59.696557 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:55:59.696582 (+ 25us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:55:59.696638 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:55:59.697406 (+ 768us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:55:59.697409 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:55:59.697411 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 14:55:59.697421 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:55:59.697442 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 14:55:59.697451 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:55:59.697524 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0927 14:55:59.697625 (+ 101us) server_negotiation.cc:299] Negotiation successful
0927 14:55:59.697666 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260927 14:55:59.699275 22603 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1de6423cda9b43efb51af5002dbbf942 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1de6423cda9b43efb51af5002dbbf942" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38915 } }
I20260927 14:55:59.699501 22603 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1de6423cda9b43efb51af5002dbbf942 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 14:55:59.699575 22603 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1de6423cda9b43efb51af5002dbbf942 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1de6423cda9b43efb51af5002dbbf942, State: Initialized, Role: FOLLOWER
I20260927 14:55:59.699998 22603 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1de6423cda9b43efb51af5002dbbf942 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1de6423cda9b43efb51af5002dbbf942" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38915 } }
I20260927 14:55:59.700147 22603 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1de6423cda9b43efb51af5002dbbf942 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 14:55:59.700209 22603 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1de6423cda9b43efb51af5002dbbf942 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 14:55:59.700299 22603 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1de6423cda9b43efb51af5002dbbf942 [term 0 FOLLOWER]: Advancing to term 1
I20260927 14:55:59.701140 22603 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1de6423cda9b43efb51af5002dbbf942 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1de6423cda9b43efb51af5002dbbf942" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38915 } }
I20260927 14:55:59.701467 22603 leader_election.cc:302] T 00000000000000000000000000000000 P 1de6423cda9b43efb51af5002dbbf942 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1de6423cda9b43efb51af5002dbbf942; no voters:
I20260927 14:55:59.701764 22603 leader_election.cc:288] T 00000000000000000000000000000000 P 1de6423cda9b43efb51af5002dbbf942 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 14:55:59.701848 22613 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1de6423cda9b43efb51af5002dbbf942 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 14:55:59.702124 22613 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1de6423cda9b43efb51af5002dbbf942 [term 1 LEADER]: Becoming Leader. State: Replica: 1de6423cda9b43efb51af5002dbbf942, State: Running, Role: LEADER
I20260927 14:55:59.702418 22613 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1de6423cda9b43efb51af5002dbbf942 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1de6423cda9b43efb51af5002dbbf942" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38915 } }
I20260927 14:55:59.702916 22603 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1de6423cda9b43efb51af5002dbbf942 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 14:55:59.703956 22614 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1de6423cda9b43efb51af5002dbbf942 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1de6423cda9b43efb51af5002dbbf942" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1de6423cda9b43efb51af5002dbbf942" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38915 } } }
I20260927 14:55:59.704205 22614 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1de6423cda9b43efb51af5002dbbf942 [sys.catalog]: This master's current role is: LEADER
I20260927 14:55:59.704533 22615 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1de6423cda9b43efb51af5002dbbf942 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1de6423cda9b43efb51af5002dbbf942. Latest consensus state: current_term: 1 leader_uuid: "1de6423cda9b43efb51af5002dbbf942" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1de6423cda9b43efb51af5002dbbf942" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38915 } } }
I20260927 14:55:59.704627 22615 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1de6423cda9b43efb51af5002dbbf942 [sys.catalog]: This master's current role is: LEADER
I20260927 14:55:59.705039 22622 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 14:55:59.707451 22622 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 14:55:59.712594 22622 catalog_manager.cc:1347] Generated new cluster ID: 1617427da0384b01a6dff8bc2e004a52
I20260927 14:55:59.712653 22622 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 14:55:59.729789 22622 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 14:55:59.730120 22622 catalog_manager.cc:1504] Loading token signing keys...
I20260927 14:55:59.739573 22622 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1de6423cda9b43efb51af5002dbbf942: Generated new TSK 0
I20260927 14:55:59.740362 22622 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 14:55:59.744825 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6936m5/build/debug/bin/kudu with pid 22535
2026-09-27T14:55:59Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1392 ms)
[----------] 18 tests from AuthTokenIssuingTest (25307 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-task6936m5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790520684602251-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 14:55:59 dist-test-slave-xk1g krb5kdc[22634](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 14:55:59 dist-test-slave-xk1g krb5kdc[22634](info): set up 2 sockets
Sep 27 14:55:59 dist-test-slave-xk1g krb5kdc[22634](info): commencing operation
krb5kdc: starting...
W20260927 14:56:00.777448 392 mini_kdc.cc:121] Time spent starting KDC: real 1.013s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6936m5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790520684602251-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6936m5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790520684602251-392-0/krb5kdc/test-user.keytab.
Sep 27 14:56:00 dist-test-slave-xk1g krb5kdc[22634](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790520960, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T14:56:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T14:56:00Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6936m5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790520684602251-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6936m5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790520684602251-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 14:56:00.896296 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6936m5/build/debug/bin/kudu
/tmp/dist-test-task6936m5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790520684602251-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790520684602251-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790520684602251-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790520684602251-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33997
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790520684602251-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35555
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:33997,127.0.98.61:40141,127.0.98.60:44353
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790520684602251-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790520684602251-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790520684602251-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 14:56:01.005272 22647 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260927 14:56:01.005577 22647 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 14:56:01.005601 22647 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 14:56:01.005641 22647 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 14:56:01.009577 22647 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 14:56:01.009632 22647 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 14:56:01.009651 22647 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 14:56:01.009667 22647 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 14:56:01.009682 22647 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 14:56:01.015049 22647 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35555
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790520684602251-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790520684602251-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:33997,127.0.98.61:40141,127.0.98.60:44353
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790520684602251-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33997
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790520684602251-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22647
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790520684602251-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:15:17 UTC on ccycloud.quasar-ledxfe.root.comops.site
build id 4676134
I20260927 14:56:01.016109 22647 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 14:56:01.017081 22647 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 14:56:01.025552 22652 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 14:56:01.025539 22655 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 14:56:01.025616 22653 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 14:56:01.025646 22647 server_base.cc:1034] running on GCE node
I20260927 14:56:01.026412 22647 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 14:56:01.027479 22647 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 14:56:01.028646 22647 hybrid_clock.cc:648] HybridClock initialized: now 1790520961028614 us; error 45 us; skew 500 ppm
Sep 27 14:56:01 dist-test-slave-xk1g krb5kdc[22634](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790520961, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 14:56:01.031318 22647 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 14:56:01.032112 22647 webserver.cc:460] Webserver started at http://127.0.98.62:46169/ using document root <none> and password file <none>
I20260927 14:56:01.032629 22647 fs_manager.cc:359] Metadata directory not provided
I20260927 14:56:01.032685 22647 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 14:56:01.032845 22647 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 14:56:01.034884 22647 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6936m5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790520684602251-392-0/minicluster-data/master-0/data/instance:
uuid: "d657d47a8f9b4632a2916b4d061d902b"
format_stamp: "Formatted at 2026-09-27 14:56:01 on dist-test-slave-xk1g"
server_key: "a30501364cefe7625c43c1ec4e374e39"
server_key_iv: "80c954b2e0b62519b470d47e13ba7d2a"
server_key_version: "encryptionkey@0"
I20260927 14:56:01.035387 22647 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6936m5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790520684602251-392-0/minicluster-data/master-0/wal/instance:
uuid: "d657d47a8f9b4632a2916b4d061d902b"
format_stamp: "Formatted at 2026-09-27 14:56:01 on dist-test-slave-xk1g"
server_key: "a30501364cefe7625c43c1ec4e374e39"
server_key_iv: "80c954b2e0b62519b470d47e13ba7d2a"
server_key_version: "encryptionkey@0"
I20260927 14:56:01.038944 22647 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260927 14:56:01.041338 22662 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 14:56:01.042613 22647 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 14:56:01.042724 22647 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790520684602251-392-0/minicluster-data/master-0/data,/tmp/dist-test-task6936m5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790520684602251-392-0/minicluster-data/master-0/wal
uuid: "d657d47a8f9b4632a2916b4d061d902b"
format_stamp: "Formatted at 2026-09-27 14:56:01 on dist-test-slave-xk1g"
server_key: "a30501364cefe7625c43c1ec4e374e39"
server_key_iv: "80c954b2e0b62519b470d47e13ba7d2a"
server_key_version: "encryptionkey@0"
I20260927 14:56:01.042811 22647 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790520684602251-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790520684602251-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790520684602251-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 14:56:01.061401 22647 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 14:56:01.064122 22647 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 14:56:01.064272 22647 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 14:56:01.071014 22647 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:33997
I20260927 14:56:01.071310 22714 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:33997 every 8 connection(s)
I20260927 14:56:01.071990 22647 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6936m5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790520684602251-392-0/minicluster-data/master-0/data/info.pb
I20260927 14:56:01.074719 22715 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 14:56:01.078289 22715 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33997 } has no permanent_uuid. Determining permanent_uuid...
I20260927 14:56:01.082049 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6936m5/build/debug/bin/kudu as pid 22647
I20260927 14:56:01.082165 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6936m5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790520684602251-392-0/minicluster-data/master-0/wal/instance
I20260927 14:56:01.082487 392 external_mini_cluster.cc:1428] Setting key 892f2b1c66c5cd487669ebc6641d6413
Sep 27 14:56:01 dist-test-slave-xk1g krb5kdc[22634](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790520961, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.61@KRBTEST.COM" created.
I20260927 14:56:01.093613 22718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:56:01.080945 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48318 (local address 127.0.98.62:33997)
0927 14:56:01.081209 (+ 264us) server_negotiation.cc:206] Beginning negotiation
0927 14:56:01.081221 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 14:56:01.081262 (+ 41us) server_negotiation.cc:407] Connection header received
0927 14:56:01.081919 (+ 657us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:56:01.081923 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:56:01.082090 (+ 167us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:56:01.082314 (+ 224us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:56:01.083173 (+ 859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:56:01.084297 (+ 1124us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:56:01.085348 (+ 1051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:56:01.086022 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:56:01.086246 (+ 224us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:56:01.088554 (+ 2308us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:56:01.088561 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:56:01.088576 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 14:56:01.088605 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:56:01.090658 (+ 2053us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:56:01.091091 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:56:01.091096 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:56:01.091097 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:56:01.091197 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:56:01.091479 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:56:01.091482 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:56:01.091484 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:56:01.091821 (+ 337us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:56:01.091965 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0927 14:56:01.092315 (+ 350us) server_negotiation.cc:299] Negotiation successful
0927 14:56:01.092638 (+ 323us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":157,"server-negotiator.queue_time_us":175,"thread_start_us":63,"threads_started":1}
I20260927 14:56:01.093613 22717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:56:01.080275 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33997 (local address 127.0.0.1:48318)
0927 14:56:01.080705 (+ 430us) negotiation.cc:107] Waiting for socket to connect
0927 14:56:01.080747 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0927 14:56:01.081614 (+ 867us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:56:01.082322 (+ 708us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:56:01.082327 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:56:01.082748 (+ 421us) client_negotiation.cc:189] Negotiated authn=SASL
0927 14:56:01.082990 (+ 242us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:56:01.083043 (+ 53us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:56:01.084456 (+ 1413us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:56:01.084459 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:56:01.085237 (+ 778us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:56:01.085247 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:56:01.086192 (+ 945us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:56:01.086196 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:56:01.086384 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:56:01.086955 (+ 571us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 14:56:01.086970 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 14:56:01.088398 (+ 1428us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 14:56:01.090761 (+ 2363us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 14:56:01.090769 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 14:56:01.090777 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0927 14:56:01.090997 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 14:56:01.091275 (+ 278us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 14:56:01.091278 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 14:56:01.091279 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 14:56:01.091375 (+ 96us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 14:56:01.091953 (+ 578us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 14:56:01.091959 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 14:56:01.092082 (+ 123us) client_negotiation.cc:769] Sending connection context
0927 14:56:01.092244 (+ 162us) client_negotiation.cc:240] Negotiation successful
0927 14:56:01.092591 (+ 347us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":255,"mutex_wait_us":89,"thread_start_us":103,"threads_started":1}
I20260927 14:56:01.096099 22715 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40141 } has no permanent_uuid. Determining permanent_uuid...
I20260927 14:56:01.096863 22717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:56:01.096588 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:40141 (local address 127.0.0.1:47530)
0927 14:56:01.096719 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0927 14:56:01.096803 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:40141: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
W20260927 14:56:01.097179 22666 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.61:40141: connect: Connection refused (error 111)
I20260927 14:56:01.097978 22717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:56:01.097771 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:40141 (local address 127.0.0.1:47532)
0927 14:56:01.097888 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0927 14:56:01.097928 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:40141: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260927 14:56:01.098135 22715 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:40141: Network error: Client connection negotiation failed: client connection to 127.0.98.61:40141: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6936m5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790520684602251-392-0/krb5kdc/kudu_127.0.98.61.keytab.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6936m5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790520684602251-392-0/krb5kdc/kudu_127.0.98.61.keytab.
I20260927 14:56:01.108126 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6936m5/build/debug/bin/kudu
/tmp/dist-test-task6936m5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790520684602251-392-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790520684602251-392-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790520684602251-392-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790520684602251-392-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:40141
--webserver_interface=127.0.98.61
--webserver_port=0
--keytab_file=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790520684602251-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35555
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:33997,127.0.98.61:40141,127.0.98.60:44353
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790520684602251-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790520684602251-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790520684602251-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 14:56:01.147380 22715 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40141 } attempt: 1
I20260927 14:56:01.148216 22717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:56:01.147913 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:40141 (local address 127.0.0.1:47538)
0927 14:56:01.148070 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0927 14:56:01.148139 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:40141: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
I20260927 14:56:01.149037 22717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:56:01.148826 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:40141 (local address 127.0.0.1:47552)
0927 14:56:01.148937 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0927 14:56:01.148984 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:40141: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260927 14:56:01.149235 22715 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:40141: Network error: Client connection negotiation failed: client connection to 127.0.98.61:40141: connect: Connection refused (error 111)
W20260927 14:56:01.215677 22722 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260927 14:56:01.215934 22722 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 14:56:01.215956 22722 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 14:56:01.216068 22722 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260927 14:56:01.217408 22715 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40141 } attempt: 2
I20260927 14:56:01.218248 22717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:56:01.217942 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:40141 (local address 127.0.0.1:47556)
0927 14:56:01.218112 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0927 14:56:01.218175 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:40141: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
I20260927 14:56:01.219009 22717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:56:01.218814 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:40141 (local address 127.0.0.1:47564)
0927 14:56:01.218915 (+ 101us) negotiation.cc:107] Waiting for socket to connect
0927 14:56:01.218958 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:40141: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260927 14:56:01.219148 22715 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:40141: Network error: Client connection negotiation failed: client connection to 127.0.98.61:40141: connect: Connection refused (error 111)
W20260927 14:56:01.220574 22722 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 14:56:01.220623 22722 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 14:56:01.220642 22722 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 14:56:01.220659 22722 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 14:56:01.220674 22722 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 14:56:01.225874 22722 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35555
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790520684602251-392-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790520684602251-392-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:33997,127.0.98.61:40141,127.0.98.60:44353
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790520684602251-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:40141
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790520684602251-392-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.98.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22722
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790520684602251-392-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:15:17 UTC on ccycloud.quasar-ledxfe.root.comops.site
build id 4676134
I20260927 14:56:01.226763 22722 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 14:56:01.227643 22722 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 14:56:01.235960 22728 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 14:56:01.235948 22727 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 14:56:01.235981 22730 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 14:56:01.236152 22722 server_base.cc:1034] running on GCE node
I20260927 14:56:01.236735 22722 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 14:56:01.237790 22722 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 14:56:01.238952 22722 hybrid_clock.cc:648] HybridClock initialized: now 1790520961238933 us; error 29 us; skew 500 ppm
Sep 27 14:56:01 dist-test-slave-xk1g krb5kdc[22634](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790520961, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 14:56:01.241528 22722 init.cc:377] Logged in from keytab as kudu/127.0.98.61@KRBTEST.COM (short username kudu)
I20260927 14:56:01.242388 22722 webserver.cc:460] Webserver started at http://127.0.98.61:36565/ using document root <none> and password file <none>
I20260927 14:56:01.242937 22722 fs_manager.cc:359] Metadata directory not provided
I20260927 14:56:01.242995 22722 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 14:56:01.243230 22722 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 14:56:01.245673 22722 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6936m5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790520684602251-392-0/minicluster-data/master-1/data/instance:
uuid: "8687c99b31a54a2c8b58d23d71bf7309"
format_stamp: "Formatted at 2026-09-27 14:56:01 on dist-test-slave-xk1g"
server_key: "67f82dcf3e62e2f4c37ca84231b2f13b"
server_key_iv: "50bf8a49242cf0066b650af3578861f3"
server_key_version: "encryptionkey@0"
I20260927 14:56:01.246330 22722 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6936m5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790520684602251-392-0/minicluster-data/master-1/wal/instance:
uuid: "8687c99b31a54a2c8b58d23d71bf7309"
format_stamp: "Formatted at 2026-09-27 14:56:01 on dist-test-slave-xk1g"
server_key: "67f82dcf3e62e2f4c37ca84231b2f13b"
server_key_iv: "50bf8a49242cf0066b650af3578861f3"
server_key_version: "encryptionkey@0"
I20260927 14:56:01.250003 22722 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260927 14:56:01.252324 22737 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 14:56:01.253580 22722 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 14:56:01.253712 22722 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790520684602251-392-0/minicluster-data/master-1/data,/tmp/dist-test-task6936m5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790520684602251-392-0/minicluster-data/master-1/wal
uuid: "8687c99b31a54a2c8b58d23d71bf7309"
format_stamp: "Formatted at 2026-09-27 14:56:01 on dist-test-slave-xk1g"
server_key: "67f82dcf3e62e2f4c37ca84231b2f13b"
server_key_iv: "50bf8a49242cf0066b650af3578861f3"
server_key_version: "encryptionkey@0"
I20260927 14:56:01.253811 22722 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790520684602251-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790520684602251-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790520684602251-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 14:56:01.263180 22722 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 14:56:01.266088 22722 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 14:56:01.266294 22722 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 14:56:01.273221 22722 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:40141
I20260927 14:56:01.273581 22789 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:40141 every 8 connection(s)
I20260927 14:56:01.274200 22722 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6936m5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790520684602251-392-0/minicluster-data/master-1/data/info.pb
I20260927 14:56:01.276865 22790 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 14:56:01.280411 22790 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33997 } has no permanent_uuid. Determining permanent_uuid...
I20260927 14:56:01.283804 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6936m5/build/debug/bin/kudu as pid 22722
I20260927 14:56:01.283900 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6936m5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790520684602251-392-0/minicluster-data/master-1/wal/instance
I20260927 14:56:01.284190 392 external_mini_cluster.cc:1428] Setting key 4dd207e51448c8dee95682681b98db11
Sep 27 14:56:01 dist-test-slave-xk1g krb5kdc[22634](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790520961, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.60@KRBTEST.COM" created.
I20260927 14:56:01.294843 22718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:56:01.282228 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48332 (local address 127.0.98.62:33997)
0927 14:56:01.282383 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0927 14:56:01.282388 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 14:56:01.283220 (+ 832us) server_negotiation.cc:407] Connection header received
0927 14:56:01.284162 (+ 942us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:56:01.284165 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:56:01.284220 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:56:01.284330 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:56:01.285960 (+ 1630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:56:01.286816 (+ 856us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:56:01.287900 (+ 1084us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:56:01.288317 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:56:01.288439 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:56:01.291290 (+ 2851us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:56:01.291293 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:56:01.291295 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 14:56:01.291319 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:56:01.292826 (+ 1507us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:56:01.293384 (+ 558us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:56:01.293387 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:56:01.293389 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:56:01.293469 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:56:01.293787 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:56:01.293790 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:56:01.293792 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:56:01.293964 (+ 172us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:56:01.294061 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0927 14:56:01.294559 (+ 498us) server_negotiation.cc:299] Negotiation successful
0927 14:56:01.294675 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260927 14:56:01.295794 22792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:56:01.282566 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33997 (local address 127.0.0.1:48332)
0927 14:56:01.283081 (+ 515us) negotiation.cc:107] Waiting for socket to connect
0927 14:56:01.283122 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0927 14:56:01.283918 (+ 796us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:56:01.284558 (+ 640us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:56:01.284567 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:56:01.285081 (+ 514us) client_negotiation.cc:189] Negotiated authn=SASL
0927 14:56:01.285777 (+ 696us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:56:01.285801 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:56:01.286939 (+ 1138us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:56:01.286941 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:56:01.287780 (+ 839us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:56:01.287788 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:56:01.288469 (+ 681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:56:01.288472 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:56:01.288718 (+ 246us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:56:01.289362 (+ 644us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 14:56:01.289378 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 14:56:01.291131 (+ 1753us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 14:56:01.292925 (+ 1794us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 14:56:01.292930 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 14:56:01.292964 (+ 34us) client_negotiation.cc:760] calling sasl_client_step()
0927 14:56:01.293272 (+ 308us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 14:56:01.293569 (+ 297us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 14:56:01.293572 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 14:56:01.293573 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 14:56:01.293689 (+ 116us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 14:56:01.294060 (+ 371us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 14:56:01.294067 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 14:56:01.294313 (+ 246us) client_negotiation.cc:769] Sending connection context
0927 14:56:01.294491 (+ 178us) client_negotiation.cc:240] Negotiation successful
0927 14:56:01.294792 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":328,"thread_start_us":124,"threads_started":1}
I20260927 14:56:01.297199 22790 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40141 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 14:56:01 dist-test-slave-xk1g krb5kdc[22634](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790520961, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260927 14:56:01.309028 22792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:56:01.297692 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:40141 (local address 127.0.0.1:47566)
0927 14:56:01.297801 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0927 14:56:01.297816 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 14:56:01.297888 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:56:01.298667 (+ 779us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:56:01.298669 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:56:01.298795 (+ 126us) client_negotiation.cc:189] Negotiated authn=SASL
0927 14:56:01.298874 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:56:01.298878 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:56:01.300378 (+ 1500us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:56:01.300380 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:56:01.300905 (+ 525us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:56:01.300911 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:56:01.301744 (+ 833us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:56:01.301746 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:56:01.301815 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:56:01.302295 (+ 480us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 14:56:01.302305 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0927 14:56:01.305553 (+ 3248us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 14:56:01.307509 (+ 1956us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 14:56:01.307514 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 14:56:01.307519 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 14:56:01.307779 (+ 260us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 14:56:01.308058 (+ 279us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 14:56:01.308076 (+ 18us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 14:56:01.308080 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 14:56:01.308175 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 14:56:01.308556 (+ 381us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 14:56:01.308559 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 14:56:01.308659 (+ 100us) client_negotiation.cc:769] Sending connection context
0927 14:56:01.308742 (+ 83us) client_negotiation.cc:240] Negotiation successful
0927 14:56:01.308840 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":45}
I20260927 14:56:01.309199 22795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:56:01.297852 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47566 (local address 127.0.98.61:40141)
0927 14:56:01.298232 (+ 380us) server_negotiation.cc:206] Beginning negotiation
0927 14:56:01.298245 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 14:56:01.298270 (+ 25us) server_negotiation.cc:407] Connection header received
0927 14:56:01.298315 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:56:01.298323 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:56:01.298526 (+ 203us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:56:01.298675 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:56:01.298982 (+ 307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:56:01.300275 (+ 1293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:56:01.301012 (+ 737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:56:01.301631 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:56:01.301749 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:56:01.305705 (+ 3956us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:56:01.305713 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:56:01.305724 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 14:56:01.305750 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:56:01.307365 (+ 1615us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:56:01.307870 (+ 505us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:56:01.307876 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:56:01.307877 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:56:01.307958 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:56:01.308250 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:56:01.308252 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:56:01.308253 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:56:01.308478 (+ 225us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:56:01.308575 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0927 14:56:01.308798 (+ 223us) server_negotiation.cc:299] Negotiation successful
0927 14:56:01.308906 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":267,"thread_start_us":171,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6936m5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790520684602251-392-0/krb5kdc/kudu_127.0.98.60.keytab.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6936m5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790520684602251-392-0/krb5kdc/kudu_127.0.98.60.keytab.
I20260927 14:56:01.310328 22715 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40141 } attempt: 3
I20260927 14:56:01.310606 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6936m5/build/debug/bin/kudu
/tmp/dist-test-task6936m5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790520684602251-392-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790520684602251-392-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790520684602251-392-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790520684602251-392-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:44353
--webserver_interface=127.0.98.60
--webserver_port=0
--keytab_file=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790520684602251-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35555
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:33997,127.0.98.61:40141,127.0.98.60:44353
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790520684602251-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790520684602251-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790520684602251-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 14:56:01.311306 22790 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44353 } has no permanent_uuid. Determining permanent_uuid...
I20260927 14:56:01.312049 22792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:56:01.311744 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:44353 (local address 127.0.0.1:40468)
0927 14:56:01.311884 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0927 14:56:01.311964 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:44353: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
W20260927 14:56:01.312256 22739 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.60:44353: connect: Connection refused (error 111)
I20260927 14:56:01.313284 22792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:56:01.312889 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:44353 (local address 127.0.0.1:40482)
0927 14:56:01.313187 (+ 298us) negotiation.cc:107] Waiting for socket to connect
0927 14:56:01.313233 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:44353: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":220}
Sep 27 14:56:01 dist-test-slave-xk1g krb5kdc[22634](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790520961, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
W20260927 14:56:01.317286 22790 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:44353: Network error: Client connection negotiation failed: client connection to 127.0.98.60:44353: connect: Connection refused (error 111)
I20260927 14:56:01.320291 22795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:56:01.310952 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47582 (local address 127.0.98.61:40141)
0927 14:56:01.311341 (+ 389us) server_negotiation.cc:206] Beginning negotiation
0927 14:56:01.311347 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 14:56:01.311361 (+ 14us) server_negotiation.cc:407] Connection header received
0927 14:56:01.311395 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:56:01.311400 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:56:01.311447 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:56:01.311528 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:56:01.312021 (+ 493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:56:01.312850 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:56:01.314673 (+ 1823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:56:01.314894 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:56:01.314978 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:56:01.317265 (+ 2287us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:56:01.317269 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:56:01.317271 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 14:56:01.317295 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:56:01.318741 (+ 1446us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:56:01.319169 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:56:01.319173 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:56:01.319175 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:56:01.319248 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:56:01.319521 (+ 273us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:56:01.319526 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:56:01.319527 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:56:01.319686 (+ 159us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:56:01.319785 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0927 14:56:01.320059 (+ 274us) server_negotiation.cc:299] Negotiation successful
0927 14:56:01.320162 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":306}
I20260927 14:56:01.320293 22717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:56:01.310770 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:40141 (local address 127.0.0.1:47582)
0927 14:56:01.310907 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0927 14:56:01.310923 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 14:56:01.311013 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:56:01.311574 (+ 561us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:56:01.311577 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:56:01.311760 (+ 183us) client_negotiation.cc:189] Negotiated authn=SASL
0927 14:56:01.311897 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:56:01.311905 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:56:01.312987 (+ 1082us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:56:01.312991 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:56:01.314520 (+ 1529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:56:01.314536 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:56:01.315026 (+ 490us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:56:01.315033 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:56:01.315177 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:56:01.315681 (+ 504us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 14:56:01.315695 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0927 14:56:01.317143 (+ 1448us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 14:56:01.318872 (+ 1729us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 14:56:01.318875 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 14:56:01.318877 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 14:56:01.319074 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 14:56:01.319346 (+ 272us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 14:56:01.319349 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 14:56:01.319351 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 14:56:01.319434 (+ 83us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 14:56:01.319777 (+ 343us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 14:56:01.319780 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 14:56:01.319877 (+ 97us) client_negotiation.cc:769] Sending connection context
0927 14:56:01.319998 (+ 121us) client_negotiation.cc:240] Negotiation successful
0927 14:56:01.320131 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":64}
I20260927 14:56:01.321386 22715 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44353 } has no permanent_uuid. Determining permanent_uuid...
I20260927 14:56:01.322028 22717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:56:01.321803 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:44353 (local address 127.0.0.1:40484)
0927 14:56:01.321911 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0927 14:56:01.321962 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:44353: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260927 14:56:01.322708 22717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:56:01.322478 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:44353 (local address 127.0.0.1:40488)
0927 14:56:01.322619 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0927 14:56:01.322659 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:44353: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88}
W20260927 14:56:01.322893 22715 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:44353: Network error: Client connection negotiation failed: client connection to 127.0.98.60:44353: connect: Connection refused (error 111)
I20260927 14:56:01.354071 22715 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44353 } attempt: 1
I20260927 14:56:01.354897 22717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:56:01.354614 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:44353 (local address 127.0.0.1:40490)
0927 14:56:01.354759 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0927 14:56:01.354820 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:44353: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260927 14:56:01.355620 22717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:56:01.355405 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:44353 (local address 127.0.0.1:40492)
0927 14:56:01.355519 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0927 14:56:01.355567 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:44353: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260927 14:56:01.355737 22715 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:44353: Network error: Client connection negotiation failed: client connection to 127.0.98.60:44353: connect: Connection refused (error 111)
I20260927 14:56:01.366539 22790 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44353 } attempt: 1
I20260927 14:56:01.367240 22792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:56:01.366955 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:44353 (local address 127.0.0.1:40504)
0927 14:56:01.367101 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0927 14:56:01.367161 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:44353: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260927 14:56:01.367914 22792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:56:01.367700 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:44353 (local address 127.0.0.1:40508)
0927 14:56:01.367808 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0927 14:56:01.367858 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:44353: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260927 14:56:01.368047 22790 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:44353: Network error: Client connection negotiation failed: client connection to 127.0.98.60:44353: connect: Connection refused (error 111)
W20260927 14:56:01.422334 22796 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260927 14:56:01.422600 22796 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 14:56:01.422670 22796 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 14:56:01.422744 22796 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 14:56:01.426870 22796 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 14:56:01.426921 22796 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 14:56:01.426940 22796 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 14:56:01.426957 22796 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 14:56:01.426972 22796 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 14:56:01.430881 22715 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44353 } attempt: 2
I20260927 14:56:01.431612 22717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:56:01.431335 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:44353 (local address 127.0.0.1:40520)
0927 14:56:01.431478 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0927 14:56:01.431541 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:44353: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260927 14:56:01.432390 22717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:56:01.432157 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:44353 (local address 127.0.0.1:40524)
0927 14:56:01.432271 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0927 14:56:01.432326 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:44353: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20260927 14:56:01.432456 22796 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35555
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790520684602251-392-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790520684602251-392-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:33997,127.0.98.61:40141,127.0.98.60:44353
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790520684602251-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:44353
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790520684602251-392-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.98.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22796
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790520684602251-392-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:15:17 UTC on ccycloud.quasar-ledxfe.root.comops.site
build id 4676134
W20260927 14:56:01.432536 22715 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:44353: Network error: Client connection negotiation failed: client connection to 127.0.98.60:44353: connect: Connection refused (error 111)
I20260927 14:56:01.433535 22796 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 14:56:01.434448 22796 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 14:56:01.436219 22790 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44353 } attempt: 2
I20260927 14:56:01.436909 22792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:56:01.436606 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:44353 (local address 127.0.0.1:40540)
0927 14:56:01.436758 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0927 14:56:01.436825 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:44353: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260927 14:56:01.437654 22792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:56:01.437415 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:44353 (local address 127.0.0.1:40546)
0927 14:56:01.437542 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0927 14:56:01.437592 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:44353: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260927 14:56:01.437788 22790 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:44353: Network error: Client connection negotiation failed: client connection to 127.0.98.60:44353: connect: Connection refused (error 111)
W20260927 14:56:01.442889 22803 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 14:56:01.442929 22802 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 14:56:01.443023 22805 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 14:56:01.443056 22796 server_base.cc:1034] running on GCE node
I20260927 14:56:01.443696 22796 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 14:56:01.444742 22796 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 14:56:01.445917 22796 hybrid_clock.cc:648] HybridClock initialized: now 1790520961445903 us; error 25 us; skew 500 ppm
Sep 27 14:56:01 dist-test-slave-xk1g krb5kdc[22634](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790520961, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 14:56:01.448663 22796 init.cc:377] Logged in from keytab as kudu/127.0.98.60@KRBTEST.COM (short username kudu)
I20260927 14:56:01.449566 22796 webserver.cc:460] Webserver started at http://127.0.98.60:37097/ using document root <none> and password file <none>
I20260927 14:56:01.450069 22796 fs_manager.cc:359] Metadata directory not provided
I20260927 14:56:01.450125 22796 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 14:56:01.450278 22796 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 14:56:01.452210 22796 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6936m5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790520684602251-392-0/minicluster-data/master-2/data/instance:
uuid: "5b9f414954354cfbaf672438294f07c5"
format_stamp: "Formatted at 2026-09-27 14:56:01 on dist-test-slave-xk1g"
server_key: "202ba0ddb9c50b4d381e5c4b7c3803f2"
server_key_iv: "73ec83395c4f9c8a0d41a7878637b0e5"
server_key_version: "encryptionkey@0"
I20260927 14:56:01.452687 22796 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6936m5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790520684602251-392-0/minicluster-data/master-2/wal/instance:
uuid: "5b9f414954354cfbaf672438294f07c5"
format_stamp: "Formatted at 2026-09-27 14:56:01 on dist-test-slave-xk1g"
server_key: "202ba0ddb9c50b4d381e5c4b7c3803f2"
server_key_iv: "73ec83395c4f9c8a0d41a7878637b0e5"
server_key_version: "encryptionkey@0"
I20260927 14:56:01.456146 22796 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260927 14:56:01.458493 22812 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 14:56:01.459663 22796 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 14:56:01.459805 22796 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790520684602251-392-0/minicluster-data/master-2/data,/tmp/dist-test-task6936m5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790520684602251-392-0/minicluster-data/master-2/wal
uuid: "5b9f414954354cfbaf672438294f07c5"
format_stamp: "Formatted at 2026-09-27 14:56:01 on dist-test-slave-xk1g"
server_key: "202ba0ddb9c50b4d381e5c4b7c3803f2"
server_key_iv: "73ec83395c4f9c8a0d41a7878637b0e5"
server_key_version: "encryptionkey@0"
I20260927 14:56:01.459913 22796 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790520684602251-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790520684602251-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790520684602251-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 14:56:01.475855 22796 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 14:56:01.478791 22796 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 14:56:01.478988 22796 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 14:56:01.486347 22796 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:44353
I20260927 14:56:01.486610 22864 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:44353 every 8 connection(s)
I20260927 14:56:01.487313 22796 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6936m5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790520684602251-392-0/minicluster-data/master-2/data/info.pb
I20260927 14:56:01.490069 22865 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 14:56:01.493561 22865 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33997 } has no permanent_uuid. Determining permanent_uuid...
I20260927 14:56:01.497321 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6936m5/build/debug/bin/kudu as pid 22796
I20260927 14:56:01.497473 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6936m5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790520684602251-392-0/minicluster-data/master-2/wal/instance
I20260927 14:56:01.497738 392 external_mini_cluster.cc:1428] Setting key 0a018af793ef216712347661561229d8
Sep 27 14:56:01 dist-test-slave-xk1g krb5kdc[22634](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790520961, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
Sep 27 14:56:01 dist-test-slave-xk1g krb5kdc[22634](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790520960, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260927 14:56:01.506910 22718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:56:01.495248 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48348 (local address 127.0.98.62:33997)
0927 14:56:01.495374 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0927 14:56:01.495378 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 14:56:01.496151 (+ 773us) server_negotiation.cc:407] Connection header received
0927 14:56:01.496969 (+ 818us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:56:01.496972 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:56:01.497016 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:56:01.497162 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:56:01.498412 (+ 1250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:56:01.498998 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:56:01.500109 (+ 1111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:56:01.500327 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:56:01.500448 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:56:01.502986 (+ 2538us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:56:01.502989 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:56:01.502992 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 14:56:01.503024 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:56:01.504474 (+ 1450us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:56:01.505002 (+ 528us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:56:01.505007 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:56:01.505009 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:56:01.505148 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:56:01.505557 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:56:01.505560 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:56:01.505562 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:56:01.505861 (+ 299us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:56:01.505976 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0927 14:56:01.506575 (+ 599us) server_negotiation.cc:299] Negotiation successful
0927 14:56:01.506709 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260927 14:56:01.507706 22867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:56:01.495596 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33997 (local address 127.0.0.1:48348)
0927 14:56:01.496003 (+ 407us) negotiation.cc:107] Waiting for socket to connect
0927 14:56:01.496044 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0927 14:56:01.496784 (+ 740us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:56:01.497306 (+ 522us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:56:01.497313 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:56:01.497734 (+ 421us) client_negotiation.cc:189] Negotiated authn=SASL
0927 14:56:01.498237 (+ 503us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:56:01.498250 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:56:01.499131 (+ 881us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:56:01.499135 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:56:01.499985 (+ 850us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:56:01.499994 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:56:01.500474 (+ 480us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:56:01.500479 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:56:01.500654 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:56:01.501273 (+ 619us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 14:56:01.501290 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 14:56:01.502853 (+ 1563us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 14:56:01.504601 (+ 1748us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 14:56:01.504607 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 14:56:01.504623 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0927 14:56:01.504887 (+ 264us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 14:56:01.505240 (+ 353us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 14:56:01.505244 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 14:56:01.505246 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 14:56:01.505445 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 14:56:01.505999 (+ 554us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 14:56:01.506008 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 14:56:01.506310 (+ 302us) client_negotiation.cc:769] Sending connection context
0927 14:56:01.506479 (+ 169us) client_negotiation.cc:240] Negotiation successful
0927 14:56:01.506778 (+ 299us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":240,"thread_start_us":105,"threads_started":1}
I20260927 14:56:01.508843 22869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:56:01.498374 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40550 (local address 127.0.98.60:44353)
0927 14:56:01.498988 (+ 614us) server_negotiation.cc:206] Beginning negotiation
0927 14:56:01.498996 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 14:56:01.499016 (+ 20us) server_negotiation.cc:407] Connection header received
0927 14:56:01.499046 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:56:01.499050 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:56:01.499188 (+ 138us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:56:01.499348 (+ 160us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:56:01.500052 (+ 704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:56:01.500777 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:56:01.501681 (+ 904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:56:01.502194 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:56:01.502305 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:56:01.505092 (+ 2787us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:56:01.505104 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:56:01.505125 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 14:56:01.505147 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:56:01.506906 (+ 1759us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:56:01.507447 (+ 541us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:56:01.507455 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:56:01.507461 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:56:01.507541 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:56:01.507822 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:56:01.507828 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:56:01.507833 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:56:01.508091 (+ 258us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:56:01.508288 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0927 14:56:01.508528 (+ 240us) server_negotiation.cc:299] Negotiation successful
0927 14:56:01.508676 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":538,"thread_start_us":148,"threads_started":1}
I20260927 14:56:01.509240 22865 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40141 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 14:56:01 dist-test-slave-xk1g krb5kdc[22634](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790520961, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
Sep 27 14:56:01 dist-test-slave-xk1g krb5kdc[22634](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790520960, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 14:56:01.517578 22867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:56:01.509746 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:40141 (local address 127.0.0.1:47592)
0927 14:56:01.509896 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0927 14:56:01.509910 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 14:56:01.509998 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:56:01.510220 (+ 222us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:56:01.510223 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:56:01.510386 (+ 163us) client_negotiation.cc:189] Negotiated authn=SASL
0927 14:56:01.510507 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:56:01.510514 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:56:01.511539 (+ 1025us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:56:01.511541 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:56:01.512066 (+ 525us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:56:01.512074 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:56:01.512527 (+ 453us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:56:01.512531 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:56:01.512634 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:56:01.513132 (+ 498us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 14:56:01.513152 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0927 14:56:01.514397 (+ 1245us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 14:56:01.516128 (+ 1731us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 14:56:01.516134 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 14:56:01.516136 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 14:56:01.516323 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 14:56:01.516600 (+ 277us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 14:56:01.516603 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 14:56:01.516605 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 14:56:01.516687 (+ 82us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 14:56:01.517059 (+ 372us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 14:56:01.517062 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 14:56:01.517179 (+ 117us) client_negotiation.cc:769] Sending connection context
0927 14:56:01.517281 (+ 102us) client_negotiation.cc:240] Negotiation successful
0927 14:56:01.517396 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":90}
I20260927 14:56:01.517578 22795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:56:01.509856 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47592 (local address 127.0.98.61:40141)
0927 14:56:01.509980 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0927 14:56:01.509984 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 14:56:01.510005 (+ 21us) server_negotiation.cc:407] Connection header received
0927 14:56:01.510081 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:56:01.510083 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:56:01.510120 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:56:01.510183 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:56:01.510661 (+ 478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:56:01.511425 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:56:01.512190 (+ 765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:56:01.512414 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:56:01.512530 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:56:01.514535 (+ 2005us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:56:01.514539 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:56:01.514542 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 14:56:01.514589 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:56:01.516021 (+ 1432us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:56:01.516419 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:56:01.516424 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:56:01.516426 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:56:01.516502 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:56:01.516774 (+ 272us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:56:01.516779 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:56:01.516780 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:56:01.516954 (+ 174us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:56:01.517068 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0927 14:56:01.517341 (+ 273us) server_negotiation.cc:299] Negotiation successful
0927 14:56:01.517443 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260927 14:56:01.518225 22865 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44353 } has no permanent_uuid. Determining permanent_uuid...
I20260927 14:56:01.519641 22718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:56:01.511039 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48358 (local address 127.0.98.62:33997)
0927 14:56:01.511146 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0927 14:56:01.511149 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 14:56:01.511159 (+ 10us) server_negotiation.cc:407] Connection header received
0927 14:56:01.511238 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:56:01.511240 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:56:01.511269 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:56:01.511329 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:56:01.511986 (+ 657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:56:01.512454 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:56:01.513268 (+ 814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:56:01.513494 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:56:01.513602 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:56:01.516146 (+ 2544us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:56:01.516149 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:56:01.516151 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 14:56:01.516182 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:56:01.517615 (+ 1433us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:56:01.518400 (+ 785us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:56:01.518403 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:56:01.518405 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:56:01.518484 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:56:01.518804 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:56:01.518810 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:56:01.518813 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:56:01.518993 (+ 180us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:56:01.519232 (+ 239us) server_negotiation.cc:1035] Waiting for connection context
0927 14:56:01.519351 (+ 119us) server_negotiation.cc:299] Negotiation successful
0927 14:56:01.519453 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
Sep 27 14:56:01 dist-test-slave-xk1g krb5kdc[22634](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790520961, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
Sep 27 14:56:01 dist-test-slave-xk1g krb5kdc[22634](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790520960, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260927 14:56:01.526302 22867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:56:01.518686 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:44353 (local address 127.0.0.1:40566)
0927 14:56:01.518766 (+ 80us) negotiation.cc:107] Waiting for socket to connect
0927 14:56:01.518781 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 14:56:01.518850 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:56:01.519075 (+ 225us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:56:01.519078 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:56:01.519235 (+ 157us) client_negotiation.cc:189] Negotiated authn=SASL
0927 14:56:01.519289 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:56:01.519294 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:56:01.520067 (+ 773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:56:01.520070 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:56:01.520644 (+ 574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:56:01.520652 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:56:01.521253 (+ 601us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:56:01.521257 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:56:01.521332 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:56:01.521737 (+ 405us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 14:56:01.521747 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0927 14:56:01.523000 (+ 1253us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 14:56:01.524887 (+ 1887us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 14:56:01.524889 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 14:56:01.524890 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 14:56:01.525145 (+ 255us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 14:56:01.525443 (+ 298us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 14:56:01.525449 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 14:56:01.525451 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 14:56:01.525538 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 14:56:01.525859 (+ 321us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 14:56:01.525862 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 14:56:01.525966 (+ 104us) client_negotiation.cc:769] Sending connection context
0927 14:56:01.526070 (+ 104us) client_negotiation.cc:240] Negotiation successful
0927 14:56:01.526160 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":24,"mutex_wait_us":2}
I20260927 14:56:01.526819 22869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:56:01.518754 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40566 (local address 127.0.98.60:44353)
0927 14:56:01.518840 (+ 86us) server_negotiation.cc:206] Beginning negotiation
0927 14:56:01.518843 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 14:56:01.518856 (+ 13us) server_negotiation.cc:407] Connection header received
0927 14:56:01.518937 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:56:01.518940 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:56:01.518984 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:56:01.519066 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:56:01.519389 (+ 323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:56:01.519962 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:56:01.520760 (+ 798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:56:01.521146 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:56:01.521248 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:56:01.523128 (+ 1880us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:56:01.523132 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:56:01.523135 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 14:56:01.523158 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:56:01.524766 (+ 1608us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:56:01.525235 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:56:01.525241 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:56:01.525245 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:56:01.525333 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:56:01.525624 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:56:01.525626 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:56:01.525627 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:56:01.525768 (+ 141us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:56:01.525837 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0927 14:56:01.526173 (+ 336us) server_negotiation.cc:299] Negotiation successful
0927 14:56:01.526286 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260927 14:56:01.528954 22790 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44353 } attempt: 3
I20260927 14:56:01.530552 22795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:56:01.520773 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47602 (local address 127.0.98.61:40141)
0927 14:56:01.520887 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0927 14:56:01.520891 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 14:56:01.520903 (+ 12us) server_negotiation.cc:407] Connection header received
0927 14:56:01.521003 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:56:01.521005 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:56:01.521036 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:56:01.521151 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:56:01.521873 (+ 722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:56:01.522489 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:56:01.523409 (+ 920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:56:01.523726 (+ 317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:56:01.523803 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:56:01.526537 (+ 2734us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:56:01.526540 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:56:01.526542 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 14:56:01.526561 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:56:01.528235 (+ 1674us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:56:01.528609 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:56:01.528613 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:56:01.528614 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:56:01.528697 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:56:01.528939 (+ 242us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:56:01.528943 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:56:01.528944 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:56:01.529261 (+ 317us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:56:01.529764 (+ 503us) server_negotiation.cc:1035] Waiting for connection context
0927 14:56:01.530312 (+ 548us) server_negotiation.cc:299] Negotiation successful
0927 14:56:01.530409 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260927 14:56:01.530790 22865 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5b9f414954354cfbaf672438294f07c5: Bootstrap starting.
I20260927 14:56:01.531679 22715 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44353 } attempt: 3
Sep 27 14:56:01 dist-test-slave-xk1g krb5kdc[22634](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790520961, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260927 14:56:01.533726 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 14:56:01.534020 22865 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5b9f414954354cfbaf672438294f07c5: Neither blocks nor log segments found. Creating new log.
I20260927 14:56:01.535202 22865 log.cc:824] T 00000000000000000000000000000000 P 5b9f414954354cfbaf672438294f07c5: Log is configured to *not* fsync() on all Append() calls
I20260927 14:56:01.534999 22689 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "d657d47a8f9b4632a2916b4d061d902b"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48358
Sep 27 14:56:01 dist-test-slave-xk1g krb5kdc[22634](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790520961, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260927 14:56:01.538280 22865 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5b9f414954354cfbaf672438294f07c5: No bootstrap required, opened a new log
I20260927 14:56:01.539381 22792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:56:01.529354 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:44353 (local address 127.0.0.1:40578)
0927 14:56:01.529472 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0927 14:56:01.529486 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 14:56:01.529571 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:56:01.529783 (+ 212us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:56:01.529786 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:56:01.529950 (+ 164us) client_negotiation.cc:189] Negotiated authn=SASL
0927 14:56:01.530015 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:56:01.530020 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:56:01.530869 (+ 849us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:56:01.530875 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:56:01.531592 (+ 717us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:56:01.531603 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:56:01.532203 (+ 600us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:56:01.532209 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:56:01.532288 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:56:01.532703 (+ 415us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 14:56:01.532717 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0927 14:56:01.534057 (+ 1340us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 14:56:01.537657 (+ 3600us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 14:56:01.537660 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 14:56:01.537662 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 14:56:01.537893 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 14:56:01.538238 (+ 345us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 14:56:01.538244 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 14:56:01.538247 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 14:56:01.538354 (+ 107us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 14:56:01.538946 (+ 592us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 14:56:01.538949 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 14:56:01.539038 (+ 89us) client_negotiation.cc:769] Sending connection context
0927 14:56:01.539117 (+ 79us) client_negotiation.cc:240] Negotiation successful
0927 14:56:01.539239 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32}
I20260927 14:56:01.539968 22869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:56:01.529368 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40578 (local address 127.0.98.60:44353)
0927 14:56:01.529490 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0927 14:56:01.529493 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 14:56:01.529536 (+ 43us) server_negotiation.cc:407] Connection header received
0927 14:56:01.529647 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:56:01.529649 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:56:01.529681 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:56:01.529768 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:56:01.530119 (+ 351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:56:01.530752 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:56:01.531745 (+ 993us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:56:01.532065 (+ 320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:56:01.532488 (+ 423us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:56:01.534203 (+ 1715us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:56:01.534207 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:56:01.534210 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 14:56:01.534237 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:56:01.537532 (+ 3295us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:56:01.538015 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:56:01.538019 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:56:01.538020 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:56:01.538101 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:56:01.538593 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:56:01.538597 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:56:01.538601 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:56:01.538827 (+ 226us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:56:01.539648 (+ 821us) server_negotiation.cc:1035] Waiting for connection context
0927 14:56:01.539719 (+ 71us) server_negotiation.cc:299] Negotiation successful
0927 14:56:01.539835 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260927 14:56:01.541533 22717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:56:01.532030 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:44353 (local address 127.0.0.1:40594)
0927 14:56:01.532193 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0927 14:56:01.532207 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 14:56:01.532277 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:56:01.533267 (+ 990us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:56:01.533270 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:56:01.533438 (+ 168us) client_negotiation.cc:189] Negotiated authn=SASL
0927 14:56:01.533535 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:56:01.533545 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:56:01.534646 (+ 1101us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:56:01.534651 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:56:01.535413 (+ 762us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:56:01.535425 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:56:01.536011 (+ 586us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:56:01.536016 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:56:01.536108 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:56:01.536503 (+ 395us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 14:56:01.536526 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0927 14:56:01.537881 (+ 1355us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 14:56:01.539930 (+ 2049us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 14:56:01.539940 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 14:56:01.539942 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 14:56:01.540219 (+ 277us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 14:56:01.540498 (+ 279us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 14:56:01.540501 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 14:56:01.540503 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 14:56:01.540605 (+ 102us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 14:56:01.541024 (+ 419us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 14:56:01.541027 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 14:56:01.541165 (+ 138us) client_negotiation.cc:769] Sending connection context
0927 14:56:01.541246 (+ 81us) client_negotiation.cc:240] Negotiation successful
0927 14:56:01.541371 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":99}
I20260927 14:56:01.542044 22870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:56:01.532603 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40594 (local address 127.0.98.60:44353)
0927 14:56:01.533015 (+ 412us) server_negotiation.cc:206] Beginning negotiation
0927 14:56:01.533025 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 14:56:01.533037 (+ 12us) server_negotiation.cc:407] Connection header received
0927 14:56:01.533093 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:56:01.533097 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:56:01.533135 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:56:01.533649 (+ 514us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:56:01.533749 (+ 100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:56:01.534502 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:56:01.535553 (+ 1051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:56:01.535889 (+ 336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:56:01.535993 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:56:01.538046 (+ 2053us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:56:01.538050 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:56:01.538052 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 14:56:01.538081 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:56:01.539834 (+ 1753us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:56:01.540301 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:56:01.540306 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:56:01.540307 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:56:01.540392 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:56:01.540721 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:56:01.540724 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:56:01.540726 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:56:01.540913 (+ 187us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:56:01.541702 (+ 789us) server_negotiation.cc:1035] Waiting for connection context
0927 14:56:01.541778 (+ 76us) server_negotiation.cc:299] Negotiation successful
0927 14:56:01.541886 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":349,"thread_start_us":101,"threads_started":1}
I20260927 14:56:01.542383 22865 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5b9f414954354cfbaf672438294f07c5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d657d47a8f9b4632a2916b4d061d902b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33997 } } peers { permanent_uuid: "8687c99b31a54a2c8b58d23d71bf7309" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40141 } } peers { permanent_uuid: "5b9f414954354cfbaf672438294f07c5" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44353 } }
I20260927 14:56:01.542640 22865 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5b9f414954354cfbaf672438294f07c5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b9f414954354cfbaf672438294f07c5, State: Initialized, Role: FOLLOWER
I20260927 14:56:01.543002 22790 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8687c99b31a54a2c8b58d23d71bf7309: Bootstrap starting.
I20260927 14:56:01.542979 22865 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5b9f414954354cfbaf672438294f07c5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d657d47a8f9b4632a2916b4d061d902b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33997 } } peers { permanent_uuid: "8687c99b31a54a2c8b58d23d71bf7309" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40141 } } peers { permanent_uuid: "5b9f414954354cfbaf672438294f07c5" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44353 } }
I20260927 14:56:01.543490 22872 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5b9f414954354cfbaf672438294f07c5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d657d47a8f9b4632a2916b4d061d902b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33997 } } peers { permanent_uuid: "8687c99b31a54a2c8b58d23d71bf7309" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40141 } } peers { permanent_uuid: "5b9f414954354cfbaf672438294f07c5" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44353 } } }
I20260927 14:56:01.543607 22872 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5b9f414954354cfbaf672438294f07c5 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 14:56:01.544351 22865 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5b9f414954354cfbaf672438294f07c5 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 14:56:01.544896 22715 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d657d47a8f9b4632a2916b4d061d902b: Bootstrap starting.
I20260927 14:56:01.545188 22790 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8687c99b31a54a2c8b58d23d71bf7309: Neither blocks nor log segments found. Creating new log.
I20260927 14:56:01.546046 22790 log.cc:824] T 00000000000000000000000000000000 P 8687c99b31a54a2c8b58d23d71bf7309: Log is configured to *not* fsync() on all Append() calls
I20260927 14:56:01.547394 22715 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d657d47a8f9b4632a2916b4d061d902b: Neither blocks nor log segments found. Creating new log.
I20260927 14:56:01.548164 22715 log.cc:824] T 00000000000000000000000000000000 P d657d47a8f9b4632a2916b4d061d902b: Log is configured to *not* fsync() on all Append() calls
I20260927 14:56:01.548218 22790 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8687c99b31a54a2c8b58d23d71bf7309: No bootstrap required, opened a new log
W20260927 14:56:01.549674 22883 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 5b9f414954354cfbaf672438294f07c5: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 14:56:01.549751 22883 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 14:56:01.547837 22796 master_runner.cc:186] Error getting master registration for 127.0.98.62:33997: OK, instance_id { permanent_uuid: "d657d47a8f9b4632a2916b4d061d902b" instance_seqno: 1790520961069101 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260927 14:56:01.551209 22796 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260927 14:56:01.551460 22715 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d657d47a8f9b4632a2916b4d061d902b: No bootstrap required, opened a new log
I20260927 14:56:01.552016 22790 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8687c99b31a54a2c8b58d23d71bf7309 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d657d47a8f9b4632a2916b4d061d902b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33997 } } peers { permanent_uuid: "8687c99b31a54a2c8b58d23d71bf7309" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40141 } } peers { permanent_uuid: "5b9f414954354cfbaf672438294f07c5" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44353 } }
I20260927 14:56:01.552227 22790 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8687c99b31a54a2c8b58d23d71bf7309 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8687c99b31a54a2c8b58d23d71bf7309, State: Initialized, Role: FOLLOWER
I20260927 14:56:01.552531 22790 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8687c99b31a54a2c8b58d23d71bf7309 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d657d47a8f9b4632a2916b4d061d902b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33997 } } peers { permanent_uuid: "8687c99b31a54a2c8b58d23d71bf7309" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40141 } } peers { permanent_uuid: "5b9f414954354cfbaf672438294f07c5" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44353 } }
I20260927 14:56:01.552989 22887 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8687c99b31a54a2c8b58d23d71bf7309 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d657d47a8f9b4632a2916b4d061d902b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33997 } } peers { permanent_uuid: "8687c99b31a54a2c8b58d23d71bf7309" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40141 } } peers { permanent_uuid: "5b9f414954354cfbaf672438294f07c5" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44353 } } }
I20260927 14:56:01.553227 22887 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8687c99b31a54a2c8b58d23d71bf7309 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 14:56:01.553947 22790 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8687c99b31a54a2c8b58d23d71bf7309 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 14:56:01.554087 22715 raft_consensus.cc:348] T 00000000000000000000000000000000 P d657d47a8f9b4632a2916b4d061d902b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d657d47a8f9b4632a2916b4d061d902b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33997 } } peers { permanent_uuid: "8687c99b31a54a2c8b58d23d71bf7309" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40141 } } peers { permanent_uuid: "5b9f414954354cfbaf672438294f07c5" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44353 } }
I20260927 14:56:01.554279 22715 raft_consensus.cc:729] T 00000000000000000000000000000000 P d657d47a8f9b4632a2916b4d061d902b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d657d47a8f9b4632a2916b4d061d902b, State: Initialized, Role: FOLLOWER
I20260927 14:56:01.554569 22715 consensus_queue.cc:260] T 00000000000000000000000000000000 P d657d47a8f9b4632a2916b4d061d902b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d657d47a8f9b4632a2916b4d061d902b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33997 } } peers { permanent_uuid: "8687c99b31a54a2c8b58d23d71bf7309" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40141 } } peers { permanent_uuid: "5b9f414954354cfbaf672438294f07c5" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44353 } }
I20260927 14:56:01.555728 22715 sys_catalog.cc:564] T 00000000000000000000000000000000 P d657d47a8f9b4632a2916b4d061d902b [sys.catalog]: configured and running, proceeding with master startup.
I20260927 14:56:01.555819 22689 raft_consensus.cc:482] T 00000000000000000000000000000000 P d657d47a8f9b4632a2916b4d061d902b [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260927 14:56:01.555903 22689 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d657d47a8f9b4632a2916b4d061d902b [term 0 FOLLOWER]: Advancing to term 1
I20260927 14:56:01.557204 22689 raft_consensus.cc:504] T 00000000000000000000000000000000 P d657d47a8f9b4632a2916b4d061d902b [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d657d47a8f9b4632a2916b4d061d902b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33997 } } peers { permanent_uuid: "8687c99b31a54a2c8b58d23d71bf7309" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40141 } } peers { permanent_uuid: "5b9f414954354cfbaf672438294f07c5" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44353 } }
I20260927 14:56:01.558830 22839 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d657d47a8f9b4632a2916b4d061d902b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "5b9f414954354cfbaf672438294f07c5"
I20260927 14:56:01.558584 22764 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d657d47a8f9b4632a2916b4d061d902b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "8687c99b31a54a2c8b58d23d71bf7309"
I20260927 14:56:01.559052 22839 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5b9f414954354cfbaf672438294f07c5 [term 0 FOLLOWER]: Advancing to term 1
I20260927 14:56:01.559166 22764 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8687c99b31a54a2c8b58d23d71bf7309 [term 0 FOLLOWER]: Advancing to term 1
I20260927 14:56:01.560032 22839 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 5b9f414954354cfbaf672438294f07c5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d657d47a8f9b4632a2916b4d061d902b in term 1.
I20260927 14:56:01.560370 22764 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 8687c99b31a54a2c8b58d23d71bf7309 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d657d47a8f9b4632a2916b4d061d902b in term 1.
I20260927 14:56:01.557226 22722 master_runner.cc:186] Error getting master registration for 127.0.98.62:33997: OK, instance_id { permanent_uuid: "d657d47a8f9b4632a2916b4d061d902b" instance_seqno: 1790520961069101 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260927 14:56:01.560627 22722 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260927 14:56:01.558149 22689 leader_election.cc:288] T 00000000000000000000000000000000 P d657d47a8f9b4632a2916b4d061d902b [CANDIDATE]: Term 1 election: Requested vote from peers 8687c99b31a54a2c8b58d23d71bf7309 (127.0.98.61:40141), 5b9f414954354cfbaf672438294f07c5 (127.0.98.60:44353)
I20260927 14:56:01.560426 22664 leader_election.cc:302] T 00000000000000000000000000000000 P d657d47a8f9b4632a2916b4d061d902b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5b9f414954354cfbaf672438294f07c5, d657d47a8f9b4632a2916b4d061d902b; no voters:
I20260927 14:56:01.561825 22911 sys_catalog.cc:455] T 00000000000000000000000000000000 P d657d47a8f9b4632a2916b4d061d902b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d657d47a8f9b4632a2916b4d061d902b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33997 } } peers { permanent_uuid: "8687c99b31a54a2c8b58d23d71bf7309" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40141 } } peers { permanent_uuid: "5b9f414954354cfbaf672438294f07c5" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44353 } } }
I20260927 14:56:01.561956 22911 sys_catalog.cc:458] T 00000000000000000000000000000000 P d657d47a8f9b4632a2916b4d061d902b [sys.catalog]: This master's current role is: FOLLOWER
I20260927 14:56:01.562047 22911 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d657d47a8f9b4632a2916b4d061d902b [term 1 FOLLOWER]: Leader election won for term 1
I20260927 14:56:01.562109 22911 raft_consensus.cc:686] T 00000000000000000000000000000000 P d657d47a8f9b4632a2916b4d061d902b [term 1 LEADER]: Becoming Leader. State: Replica: d657d47a8f9b4632a2916b4d061d902b, State: Running, Role: LEADER
I20260927 14:56:01.562695 22911 consensus_queue.cc:237] T 00000000000000000000000000000000 P d657d47a8f9b4632a2916b4d061d902b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d657d47a8f9b4632a2916b4d061d902b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33997 } } peers { permanent_uuid: "8687c99b31a54a2c8b58d23d71bf7309" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40141 } } peers { permanent_uuid: "5b9f414954354cfbaf672438294f07c5" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44353 } }
I20260927 14:56:01.564456 22911 sys_catalog.cc:455] T 00000000000000000000000000000000 P d657d47a8f9b4632a2916b4d061d902b [sys.catalog]: SysCatalogTable state changed. Reason: New leader d657d47a8f9b4632a2916b4d061d902b. Latest consensus state: current_term: 1 leader_uuid: "d657d47a8f9b4632a2916b4d061d902b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d657d47a8f9b4632a2916b4d061d902b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33997 } } peers { permanent_uuid: "8687c99b31a54a2c8b58d23d71bf7309" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40141 } } peers { permanent_uuid: "5b9f414954354cfbaf672438294f07c5" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44353 } } }
I20260927 14:56:01.564579 22911 sys_catalog.cc:458] T 00000000000000000000000000000000 P d657d47a8f9b4632a2916b4d061d902b [sys.catalog]: This master's current role is: LEADER
W20260927 14:56:01.564612 22899 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 8687c99b31a54a2c8b58d23d71bf7309: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 14:56:01.564683 22899 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20260927 14:56:01.565693 22668 tablet.cc:2369] T 00000000000000000000000000000000 P d657d47a8f9b4632a2916b4d061d902b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 14:56:01.566663 22918 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 14:56:01.567471 22647 master_runner.cc:263] No leader master found from master d657d47a8f9b4632a2916b4d061d902b
I20260927 14:56:01.567535 22647 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260927 14:56:01.569761 22918 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 14:56:01.577337 22718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:56:01.567271 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48370 (local address 127.0.98.62:33997)
0927 14:56:01.567378 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0927 14:56:01.567383 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 14:56:01.567829 (+ 446us) server_negotiation.cc:407] Connection header received
0927 14:56:01.568018 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:56:01.568021 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:56:01.568068 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:56:01.568119 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:56:01.569165 (+ 1046us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:56:01.569850 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:56:01.571285 (+ 1435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:56:01.571601 (+ 316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:56:01.571688 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:56:01.573717 (+ 2029us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:56:01.573721 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:56:01.573723 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 14:56:01.573747 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:56:01.575510 (+ 1763us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:56:01.576010 (+ 500us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:56:01.576013 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:56:01.576015 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:56:01.576098 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:56:01.576380 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:56:01.576384 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:56:01.576386 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:56:01.576577 (+ 191us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:56:01.576694 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0927 14:56:01.577044 (+ 350us) server_negotiation.cc:299] Negotiation successful
0927 14:56:01.577185 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260927 14:56:01.588706 22795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:56:01.569419 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47608 (local address 127.0.98.61:40141)
0927 14:56:01.569536 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0927 14:56:01.569541 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 14:56:01.571977 (+ 2436us) server_negotiation.cc:407] Connection header received
0927 14:56:01.572162 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:56:01.572170 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:56:01.572226 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:56:01.572275 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:56:01.574799 (+ 2524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:56:01.575615 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:56:01.578476 (+ 2861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:56:01.578768 (+ 292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:56:01.578830 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:56:01.581647 (+ 2817us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:56:01.581651 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:56:01.581653 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 14:56:01.581680 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:56:01.583367 (+ 1687us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:56:01.586169 (+ 2802us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:56:01.586173 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:56:01.586175 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:56:01.586268 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:56:01.587012 (+ 744us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:56:01.587015 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:56:01.587017 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:56:01.587264 (+ 247us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:56:01.587380 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0927 14:56:01.588403 (+ 1023us) server_negotiation.cc:299] Negotiation successful
0927 14:56:01.588538 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260927 14:56:01.589643 22870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:56:01.570139 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40596 (local address 127.0.98.60:44353)
0927 14:56:01.570246 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0927 14:56:01.570250 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 14:56:01.573569 (+ 3319us) server_negotiation.cc:407] Connection header received
0927 14:56:01.573740 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:56:01.573743 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:56:01.573795 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:56:01.573850 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:56:01.576720 (+ 2870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:56:01.577470 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:56:01.579232 (+ 1762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:56:01.579662 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:56:01.579754 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:56:01.583879 (+ 4125us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:56:01.583883 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:56:01.583886 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 14:56:01.583914 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:56:01.585634 (+ 1720us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:56:01.586582 (+ 948us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:56:01.586585 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:56:01.586587 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:56:01.586671 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:56:01.587687 (+ 1016us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:56:01.587690 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:56:01.587692 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:56:01.587922 (+ 230us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:56:01.588040 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0927 14:56:01.589250 (+ 1210us) server_negotiation.cc:299] Negotiation successful
0927 14:56:01.589487 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260927 14:56:01.594389 22764 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 8687c99b31a54a2c8b58d23d71bf7309 [term 1 FOLLOWER]: Refusing update from remote peer d657d47a8f9b4632a2916b4d061d902b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 14:56:01.594398 22839 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 5b9f414954354cfbaf672438294f07c5 [term 1 FOLLOWER]: Refusing update from remote peer d657d47a8f9b4632a2916b4d061d902b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 14:56:01.595067 22911 consensus_queue.cc:1035] T 00000000000000000000000000000000 P d657d47a8f9b4632a2916b4d061d902b [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b9f414954354cfbaf672438294f07c5" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44353 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 14:56:01.595574 22892 consensus_queue.cc:1035] T 00000000000000000000000000000000 P d657d47a8f9b4632a2916b4d061d902b [LEADER]: Connected to new peer: Peer: permanent_uuid: "8687c99b31a54a2c8b58d23d71bf7309" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40141 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 14:56:01.598960 22887 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8687c99b31a54a2c8b58d23d71bf7309 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d657d47a8f9b4632a2916b4d061d902b. Latest consensus state: current_term: 1 leader_uuid: "d657d47a8f9b4632a2916b4d061d902b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d657d47a8f9b4632a2916b4d061d902b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33997 } } peers { permanent_uuid: "8687c99b31a54a2c8b58d23d71bf7309" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40141 } } peers { permanent_uuid: "5b9f414954354cfbaf672438294f07c5" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44353 } } }
I20260927 14:56:01.598982 22872 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5b9f414954354cfbaf672438294f07c5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d657d47a8f9b4632a2916b4d061d902b. Latest consensus state: current_term: 1 leader_uuid: "d657d47a8f9b4632a2916b4d061d902b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d657d47a8f9b4632a2916b4d061d902b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33997 } } peers { permanent_uuid: "8687c99b31a54a2c8b58d23d71bf7309" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40141 } } peers { permanent_uuid: "5b9f414954354cfbaf672438294f07c5" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44353 } } }
I20260927 14:56:01.599092 22872 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5b9f414954354cfbaf672438294f07c5 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 14:56:01.599094 22887 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8687c99b31a54a2c8b58d23d71bf7309 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 14:56:01.605288 22892 sys_catalog.cc:455] T 00000000000000000000000000000000 P d657d47a8f9b4632a2916b4d061d902b [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "d657d47a8f9b4632a2916b4d061d902b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d657d47a8f9b4632a2916b4d061d902b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33997 } } peers { permanent_uuid: "8687c99b31a54a2c8b58d23d71bf7309" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40141 } } peers { permanent_uuid: "5b9f414954354cfbaf672438294f07c5" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44353 } } }
I20260927 14:56:01.605422 22892 sys_catalog.cc:458] T 00000000000000000000000000000000 P d657d47a8f9b4632a2916b4d061d902b [sys.catalog]: This master's current role is: LEADER
I20260927 14:56:01.608078 22872 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5b9f414954354cfbaf672438294f07c5 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "d657d47a8f9b4632a2916b4d061d902b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d657d47a8f9b4632a2916b4d061d902b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33997 } } peers { permanent_uuid: "8687c99b31a54a2c8b58d23d71bf7309" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40141 } } peers { permanent_uuid: "5b9f414954354cfbaf672438294f07c5" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44353 } } }
I20260927 14:56:01.608197 22872 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5b9f414954354cfbaf672438294f07c5 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 14:56:01.609263 22924 mvcc.cc:205] Tried to move back new op lower bound from 7333973858640367616 to 7333973858564435968. Current Snapshot: MvccSnapshot[applied={T|T < 7333973858640367616}]
I20260927 14:56:01.614591 22918 catalog_manager.cc:1347] Generated new cluster ID: 429ee1cda920416892b15d645e5908b3
I20260927 14:56:01.614665 22918 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 14:56:01.615486 22892 sys_catalog.cc:455] T 00000000000000000000000000000000 P d657d47a8f9b4632a2916b4d061d902b [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "d657d47a8f9b4632a2916b4d061d902b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d657d47a8f9b4632a2916b4d061d902b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33997 } } peers { permanent_uuid: "8687c99b31a54a2c8b58d23d71bf7309" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40141 } } peers { permanent_uuid: "5b9f414954354cfbaf672438294f07c5" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44353 } } }
I20260927 14:56:01.615608 22892 sys_catalog.cc:458] T 00000000000000000000000000000000 P d657d47a8f9b4632a2916b4d061d902b [sys.catalog]: This master's current role is: LEADER
I20260927 14:56:01.617576 22887 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8687c99b31a54a2c8b58d23d71bf7309 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "d657d47a8f9b4632a2916b4d061d902b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d657d47a8f9b4632a2916b4d061d902b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33997 } } peers { permanent_uuid: "8687c99b31a54a2c8b58d23d71bf7309" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40141 } } peers { permanent_uuid: "5b9f414954354cfbaf672438294f07c5" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44353 } } }
I20260927 14:56:01.617699 22887 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8687c99b31a54a2c8b58d23d71bf7309 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 14:56:01.619045 22925 mvcc.cc:205] Tried to move back new op lower bound from 7333973858640367616 to 7333973858564435968. Current Snapshot: MvccSnapshot[applied={T|T < 7333973858640367616}]
I20260927 14:56:01.640200 22918 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 14:56:01.641455 22918 catalog_manager.cc:1504] Loading token signing keys...
I20260927 14:56:01.660799 22918 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d657d47a8f9b4632a2916b4d061d902b: Generated new TSK 0
I20260927 14:56:01.661739 22918 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260927 14:56:01.691195 22870 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 14:56:01.687911 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40600 (local address 127.0.98.60:44353)
0927 14:56:01.688038 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0927 14:56:01.688043 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 14:56:01.690125 (+ 2082us) server_negotiation.cc:407] Connection header received
0927 14:56:01.690322 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:56:01.690326 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:56:01.690375 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:56:01.690429 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:56:01.691057 (+ 628us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40600: BlockingRecv error: recv got EOF from 127.0.0.1:40600 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
W20260927 14:56:01.692497 22718 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 14:56:01.688360 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48378 (local address 127.0.98.62:33997)
0927 14:56:01.688470 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0927 14:56:01.688474 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 14:56:01.691647 (+ 3173us) server_negotiation.cc:407] Connection header received
0927 14:56:01.691811 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:56:01.691814 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:56:01.691861 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:56:01.691922 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:56:01.692380 (+ 458us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48378: BlockingRecv error: recv got EOF from 127.0.0.1:48378 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
W20260927 14:56:01.693801 22795 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 14:56:01.688765 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47610 (local address 127.0.98.61:40141)
0927 14:56:01.688886 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0927 14:56:01.688891 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 14:56:01.692946 (+ 4055us) server_negotiation.cc:407] Connection header received
0927 14:56:01.693134 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:56:01.693137 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:56:01.693184 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:56:01.693234 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:56:01.693690 (+ 456us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47610: BlockingRecv error: recv got EOF from 127.0.0.1:47610 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
W20260927 14:56:01.698146 22795 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 14:56:01.696657 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47616 (local address 127.0.98.61:40141)
0927 14:56:01.696753 (+ 96us) server_negotiation.cc:206] Beginning negotiation
0927 14:56:01.696757 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 14:56:01.697256 (+ 499us) server_negotiation.cc:407] Connection header received
0927 14:56:01.697461 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:56:01.697464 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:56:01.697531 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:56:01.697577 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:56:01.698049 (+ 472us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47616: BlockingRecv error: recv got EOF from 127.0.0.1:47616 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
W20260927 14:56:01.703079 22795 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 14:56:01.701687 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47620 (local address 127.0.98.61:40141)
0927 14:56:01.701785 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0927 14:56:01.701789 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 14:56:01.702248 (+ 459us) server_negotiation.cc:407] Connection header received
0927 14:56:01.702420 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:56:01.702423 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:56:01.702469 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:56:01.702516 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:56:01.702982 (+ 466us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47620: BlockingRecv error: recv got EOF from 127.0.0.1:47620 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
W20260927 14:56:01.709633 22795 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 14:56:01.708257 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47624 (local address 127.0.98.61:40141)
0927 14:56:01.708349 (+ 92us) server_negotiation.cc:206] Beginning negotiation
0927 14:56:01.708353 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 14:56:01.708791 (+ 438us) server_negotiation.cc:407] Connection header received
0927 14:56:01.708974 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:56:01.708977 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:56:01.709021 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:56:01.709093 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:56:01.709542 (+ 449us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47624: BlockingRecv error: recv got EOF from 127.0.0.1:47624 (error 108)
Metrics: {"server-negotiator.queue_time_us":35}
W20260927 14:56:01.717139 22795 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 14:56:01.715685 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47638 (local address 127.0.98.61:40141)
0927 14:56:01.715803 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0927 14:56:01.715807 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 14:56:01.716258 (+ 451us) server_negotiation.cc:407] Connection header received
0927 14:56:01.716423 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:56:01.716426 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:56:01.716471 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:56:01.716518 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:56:01.717012 (+ 494us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47638: BlockingRecv error: recv got EOF from 127.0.0.1:47638 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
W20260927 14:56:01.729406 22795 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 14:56:01.727571 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47650 (local address 127.0.98.61:40141)
0927 14:56:01.727694 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0927 14:56:01.727698 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 14:56:01.728327 (+ 629us) server_negotiation.cc:407] Connection header received
0927 14:56:01.728535 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:56:01.728539 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:56:01.728597 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:56:01.728649 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:56:01.729288 (+ 639us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47650: BlockingRecv error: recv got EOF from 127.0.0.1:47650 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
W20260927 14:56:01.750049 22795 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 14:56:01.747924 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47656 (local address 127.0.98.61:40141)
0927 14:56:01.748054 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0927 14:56:01.748058 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 14:56:01.748680 (+ 622us) server_negotiation.cc:407] Connection header received
0927 14:56:01.748944 (+ 264us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:56:01.748947 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:56:01.749007 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:56:01.749097 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:56:01.749928 (+ 831us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47656: BlockingRecv error: recv got EOF from 127.0.0.1:47656 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
W20260927 14:56:01.785704 22795 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 14:56:01.784366 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47662 (local address 127.0.98.61:40141)
0927 14:56:01.784490 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0927 14:56:01.784494 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 14:56:01.784794 (+ 300us) server_negotiation.cc:407] Connection header received
0927 14:56:01.784966 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:56:01.784970 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:56:01.785025 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:56:01.785127 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:56:01.785584 (+ 457us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47662: BlockingRecv error: recv got EOF from 127.0.0.1:47662 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
W20260927 14:56:01.853022 22795 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 14:56:01.851777 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47668 (local address 127.0.98.61:40141)
0927 14:56:01.851938 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0927 14:56:01.851943 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 14:56:01.852134 (+ 191us) server_negotiation.cc:407] Connection header received
0927 14:56:01.852263 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:56:01.852265 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:56:01.852325 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:56:01.852410 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:56:01.852879 (+ 469us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47668: BlockingRecv error: recv got EOF from 127.0.0.1:47668 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
W20260927 14:56:01.984292 22795 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 14:56:01.983089 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47674 (local address 127.0.98.61:40141)
0927 14:56:01.983253 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0927 14:56:01.983258 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 14:56:01.983398 (+ 140us) server_negotiation.cc:407] Connection header received
0927 14:56:01.983574 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:56:01.983577 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:56:01.983635 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:56:01.983725 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:56:01.984176 (+ 451us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47674: BlockingRecv error: recv got EOF from 127.0.0.1:47674 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
W20260927 14:56:02.243494 22795 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 14:56:02.242492 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47678 (local address 127.0.98.61:40141)
0927 14:56:02.242629 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0927 14:56:02.242632 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 14:56:02.242730 (+ 98us) server_negotiation.cc:407] Connection header received
0927 14:56:02.242868 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:56:02.242870 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:56:02.242916 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:56:02.243004 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:56:02.243393 (+ 389us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47678: BlockingRecv error: recv got EOF from 127.0.0.1:47678 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
I20260927 14:56:02.550971 22883 catalog_manager.cc:1259] Loaded cluster ID: 429ee1cda920416892b15d645e5908b3
I20260927 14:56:02.551071 22883 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 5b9f414954354cfbaf672438294f07c5: loading cluster ID for follower catalog manager: success
I20260927 14:56:02.552675 22883 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 5b9f414954354cfbaf672438294f07c5: acquiring CA information for follower catalog manager: success
I20260927 14:56:02.553925 22883 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 5b9f414954354cfbaf672438294f07c5: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260927 14:56:02.565832 22899 catalog_manager.cc:1259] Loaded cluster ID: 429ee1cda920416892b15d645e5908b3
I20260927 14:56:02.565894 22899 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 8687c99b31a54a2c8b58d23d71bf7309: loading cluster ID for follower catalog manager: success
I20260927 14:56:02.567469 22899 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 8687c99b31a54a2c8b58d23d71bf7309: acquiring CA information for follower catalog manager: success
I20260927 14:56:02.568596 22899 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 8687c99b31a54a2c8b58d23d71bf7309: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260927 14:56:02.761659 22996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:56:02.757658 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47686 (local address 127.0.98.61:40141)
0927 14:56:02.757911 (+ 253us) server_negotiation.cc:206] Beginning negotiation
0927 14:56:02.757916 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 14:56:02.758034 (+ 118us) server_negotiation.cc:407] Connection header received
0927 14:56:02.758203 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:56:02.758206 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:56:02.758338 (+ 132us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:56:02.758457 (+ 119us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 14:56:02.758729 (+ 272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:56:02.759262 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:56:02.760132 (+ 870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:56:02.760355 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:56:02.760460 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:56:02.760807 (+ 347us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 14:56:02.761272 (+ 465us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 14:56:02.761408 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0927 14:56:02.761445 (+ 37us) server_negotiation.cc:299] Negotiation successful
0927 14:56:02.761499 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":166,"thread_start_us":69,"threads_started":1}
I20260927 14:56:02.768514 23002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:56:02.764776 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40604 (local address 127.0.98.60:44353)
0927 14:56:02.765026 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0927 14:56:02.765030 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 14:56:02.765045 (+ 15us) server_negotiation.cc:407] Connection header received
0927 14:56:02.765224 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:56:02.765232 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:56:02.765348 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:56:02.765467 (+ 119us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 14:56:02.765686 (+ 219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:56:02.766201 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:56:02.767050 (+ 849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:56:02.767282 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:56:02.767390 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:56:02.767694 (+ 304us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 14:56:02.768125 (+ 431us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 14:56:02.768257 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0927 14:56:02.768298 (+ 41us) server_negotiation.cc:299] Negotiation successful
0927 14:56:02.768354 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":149,"thread_start_us":71,"threads_started":1}
I20260927 14:56:02.775020 23002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:56:02.771487 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40608 (local address 127.0.98.60:44353)
0927 14:56:02.771612 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0927 14:56:02.771616 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 14:56:02.771920 (+ 304us) server_negotiation.cc:407] Connection header received
0927 14:56:02.772082 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:56:02.772085 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:56:02.772128 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:56:02.772171 (+ 43us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 14:56:02.772575 (+ 404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:56:02.773134 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:56:02.774031 (+ 897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:56:02.774244 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:56:02.774348 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:56:02.774582 (+ 234us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 14:56:02.774647 (+ 65us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 14:56:02.774721 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0927 14:56:02.774832 (+ 111us) server_negotiation.cc:299] Negotiation successful
0927 14:56:02.774884 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260927 14:56:02.775480 22996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:56:02.771550 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47690 (local address 127.0.98.61:40141)
0927 14:56:02.771703 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0927 14:56:02.771707 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 14:56:02.771809 (+ 102us) server_negotiation.cc:407] Connection header received
0927 14:56:02.771924 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:56:02.771927 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:56:02.771978 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:56:02.772062 (+ 84us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 14:56:02.772270 (+ 208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:56:02.773031 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:56:02.774270 (+ 1239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:56:02.774589 (+ 319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:56:02.774663 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:56:02.775027 (+ 364us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 14:56:02.775101 (+ 74us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 14:56:02.775148 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0927 14:56:02.775315 (+ 167us) server_negotiation.cc:299] Negotiation successful
0927 14:56:02.775368 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260927 14:56:02.776279 23009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:56:02.771327 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48386 (local address 127.0.98.62:33997)
0927 14:56:02.771697 (+ 370us) server_negotiation.cc:206] Beginning negotiation
0927 14:56:02.771701 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 14:56:02.771953 (+ 252us) server_negotiation.cc:407] Connection header received
0927 14:56:02.772055 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:56:02.772061 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:56:02.772215 (+ 154us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:56:02.772329 (+ 114us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 14:56:02.772638 (+ 309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:56:02.773415 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:56:02.774490 (+ 1075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:56:02.774833 (+ 343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:56:02.774934 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:56:02.775330 (+ 396us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 14:56:02.775767 (+ 437us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 14:56:02.775854 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0927 14:56:02.776127 (+ 273us) server_negotiation.cc:299] Negotiation successful
0927 14:56:02.776176 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":282,"thread_start_us":61,"threads_started":1}
I20260927 14:56:02.782595 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6936m5/build/debug/bin/kudu with pid 22647
I20260927 14:56:02.788830 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6936m5/build/debug/bin/kudu with pid 22722
I20260927 14:56:02.794651 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6936m5/build/debug/bin/kudu with pid 22796
2026-09-27T14:56:02Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (3051 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-task6936m5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790520684602251-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 14:56:02 dist-test-slave-xk1g krb5kdc[23013](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 14:56:02 dist-test-slave-xk1g krb5kdc[23013](info): set up 2 sockets
Sep 27 14:56:02 dist-test-slave-xk1g krb5kdc[23013](info): commencing operation
krb5kdc: starting...
W20260927 14:56:03.825704 392 mini_kdc.cc:121] Time spent starting KDC: real 1.010s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6936m5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790520684602251-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6936m5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790520684602251-392-0/krb5kdc/test-user.keytab.
Sep 27 14:56:03 dist-test-slave-xk1g krb5kdc[23013](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790520963, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T14:56:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T14:56:03Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6936m5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790520684602251-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6936m5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790520684602251-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 14:56:03.945701 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6936m5/build/debug/bin/kudu
/tmp/dist-test-task6936m5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790520684602251-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790520684602251-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790520684602251-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790520684602251-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45019
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790520684602251-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44519
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:45019,127.0.98.61:37771,127.0.98.60:37403
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790520684602251-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790520684602251-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790520684602251-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 14:56:04.055087 23026 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260927 14:56:04.055403 23026 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 14:56:04.055471 23026 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 14:56:04.055550 23026 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 14:56:04.059540 23026 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 14:56:04.059616 23026 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 14:56:04.059654 23026 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 14:56:04.059705 23026 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 14:56:04.059743 23026 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 14:56:04.064903 23026 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44519
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790520684602251-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790520684602251-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:45019,127.0.98.61:37771,127.0.98.60:37403
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790520684602251-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45019
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790520684602251-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23026
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790520684602251-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:15:17 UTC on ccycloud.quasar-ledxfe.root.comops.site
build id 4676134
I20260927 14:56:04.065936 23026 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 14:56:04.066922 23026 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 14:56:04.074998 23034 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 14:56:04.074971 23031 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 14:56:04.074983 23032 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 14:56:04.075157 23026 server_base.cc:1034] running on GCE node
I20260927 14:56:04.075799 23026 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 14:56:04.076828 23026 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 14:56:04.078009 23026 hybrid_clock.cc:648] HybridClock initialized: now 1790520964077990 us; error 32 us; skew 500 ppm
Sep 27 14:56:04 dist-test-slave-xk1g krb5kdc[23013](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790520964, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 14:56:04.080621 23026 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 14:56:04.081544 23026 webserver.cc:460] Webserver started at http://127.0.98.62:43935/ using document root <none> and password file <none>
I20260927 14:56:04.082087 23026 fs_manager.cc:359] Metadata directory not provided
I20260927 14:56:04.082175 23026 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 14:56:04.082387 23026 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 14:56:04.084337 23026 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6936m5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790520684602251-392-0/minicluster-data/master-0/data/instance:
uuid: "ebd09ac4557a413dba8bcbfb01fcc73e"
format_stamp: "Formatted at 2026-09-27 14:56:04 on dist-test-slave-xk1g"
server_key: "8d7bb347d1dff040fe78a19be6679eb9"
server_key_iv: "82d67c77869b855c4e9af6b863524114"
server_key_version: "encryptionkey@0"
I20260927 14:56:04.084913 23026 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6936m5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790520684602251-392-0/minicluster-data/master-0/wal/instance:
uuid: "ebd09ac4557a413dba8bcbfb01fcc73e"
format_stamp: "Formatted at 2026-09-27 14:56:04 on dist-test-slave-xk1g"
server_key: "8d7bb347d1dff040fe78a19be6679eb9"
server_key_iv: "82d67c77869b855c4e9af6b863524114"
server_key_version: "encryptionkey@0"
I20260927 14:56:04.088487 23026 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260927 14:56:04.090893 23041 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 14:56:04.092131 23026 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 14:56:04.092267 23026 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790520684602251-392-0/minicluster-data/master-0/data,/tmp/dist-test-task6936m5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790520684602251-392-0/minicluster-data/master-0/wal
uuid: "ebd09ac4557a413dba8bcbfb01fcc73e"
format_stamp: "Formatted at 2026-09-27 14:56:04 on dist-test-slave-xk1g"
server_key: "8d7bb347d1dff040fe78a19be6679eb9"
server_key_iv: "82d67c77869b855c4e9af6b863524114"
server_key_version: "encryptionkey@0"
I20260927 14:56:04.092377 23026 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790520684602251-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790520684602251-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790520684602251-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 14:56:04.116253 23026 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 14:56:04.119120 23026 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 14:56:04.119329 23026 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 14:56:04.126235 23026 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:45019
I20260927 14:56:04.126528 23093 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:45019 every 8 connection(s)
I20260927 14:56:04.127197 23026 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6936m5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790520684602251-392-0/minicluster-data/master-0/data/info.pb
I20260927 14:56:04.129842 23094 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 14:56:04.131896 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6936m5/build/debug/bin/kudu as pid 23026
I20260927 14:56:04.132020 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6936m5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790520684602251-392-0/minicluster-data/master-0/wal/instance
I20260927 14:56:04.132277 392 external_mini_cluster.cc:1428] Setting key a751996dfbf5da6ad4528bb1cc4db493
I20260927 14:56:04.133723 23094 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45019 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.0.98.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.61@KRBTEST.COM" created.
Sep 27 14:56:04 dist-test-slave-xk1g krb5kdc[23013](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790520964, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 14:56:04.148887 23097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:56:04.135988 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54872 (local address 127.0.98.62:45019)
0927 14:56:04.136389 (+ 401us) server_negotiation.cc:206] Beginning negotiation
0927 14:56:04.136411 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0927 14:56:04.136702 (+ 291us) server_negotiation.cc:407] Connection header received
0927 14:56:04.137761 (+ 1059us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:56:04.137765 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:56:04.137957 (+ 192us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:56:04.138137 (+ 180us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:56:04.138954 (+ 817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:56:04.139883 (+ 929us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:56:04.140849 (+ 966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:56:04.141274 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:56:04.141436 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:56:04.144064 (+ 2628us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:56:04.144069 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:56:04.144079 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 14:56:04.144100 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:56:04.145700 (+ 1600us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:56:04.146401 (+ 701us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:56:04.146407 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:56:04.146408 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:56:04.146511 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:56:04.146902 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:56:04.146905 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:56:04.146906 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:56:04.147308 (+ 402us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:56:04.147412 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0927 14:56:04.147865 (+ 453us) server_negotiation.cc:299] Negotiation successful
0927 14:56:04.148128 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":265,"thread_start_us":132,"threads_started":1}
I20260927 14:56:04.148895 23098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:56:04.135989 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45019 (local address 127.0.0.1:54872)
0927 14:56:04.136563 (+ 574us) negotiation.cc:107] Waiting for socket to connect
0927 14:56:04.136598 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0927 14:56:04.137467 (+ 869us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:56:04.138133 (+ 666us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:56:04.138138 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:56:04.138580 (+ 442us) client_negotiation.cc:189] Negotiated authn=SASL
0927 14:56:04.138812 (+ 232us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:56:04.138826 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:56:04.140003 (+ 1177us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:56:04.140006 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:56:04.140727 (+ 721us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:56:04.140736 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:56:04.141421 (+ 685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:56:04.141425 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:56:04.141533 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:56:04.142132 (+ 599us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 14:56:04.142148 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 14:56:04.143900 (+ 1752us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 14:56:04.145807 (+ 1907us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 14:56:04.145819 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 14:56:04.145832 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0927 14:56:04.146162 (+ 330us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 14:56:04.146617 (+ 455us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 14:56:04.146622 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 14:56:04.146625 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 14:56:04.146788 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 14:56:04.147425 (+ 637us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 14:56:04.147435 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 14:56:04.147610 (+ 175us) client_negotiation.cc:769] Sending connection context
0927 14:56:04.147822 (+ 212us) client_negotiation.cc:240] Negotiation successful
0927 14:56:04.148208 (+ 386us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":427,"mutex_wait_us":67,"spinlock_wait_cycles":35712,"thread_start_us":140,"threads_started":1}
I20260927 14:56:04.151360 23094 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37771 } has no permanent_uuid. Determining permanent_uuid...
I20260927 14:56:04.152089 23098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:56:04.151824 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:37771 (local address 127.0.0.1:53888)
0927 14:56:04.151925 (+ 101us) negotiation.cc:107] Waiting for socket to connect
0927 14:56:04.151990 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:37771: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260927 14:56:04.152297 23042 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.61:37771: connect: Connection refused (error 111)
I20260927 14:56:04.153031 23098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:56:04.152860 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:37771 (local address 127.0.0.1:53902)
0927 14:56:04.152935 (+ 75us) negotiation.cc:107] Waiting for socket to connect
0927 14:56:04.152976 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:37771: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":22}
W20260927 14:56:04.153194 23094 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:37771: Network error: Client connection negotiation failed: client connection to 127.0.98.61:37771: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6936m5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790520684602251-392-0/krb5kdc/kudu_127.0.98.61.keytab.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6936m5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790520684602251-392-0/krb5kdc/kudu_127.0.98.61.keytab.
I20260927 14:56:04.158242 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6936m5/build/debug/bin/kudu
/tmp/dist-test-task6936m5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790520684602251-392-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790520684602251-392-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790520684602251-392-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790520684602251-392-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:37771
--webserver_interface=127.0.98.61
--webserver_port=0
--keytab_file=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790520684602251-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44519
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:45019,127.0.98.61:37771,127.0.98.60:37403
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790520684602251-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790520684602251-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790520684602251-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 14:56:04.202425 23094 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37771 } attempt: 1
I20260927 14:56:04.203277 23098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:56:04.202990 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:37771 (local address 127.0.0.1:53908)
0927 14:56:04.203135 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0927 14:56:04.203198 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:37771: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260927 14:56:04.203969 23098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:56:04.203759 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:37771 (local address 127.0.0.1:53922)
0927 14:56:04.203864 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0927 14:56:04.203915 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:37771: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260927 14:56:04.204100 23094 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:37771: Network error: Client connection negotiation failed: client connection to 127.0.98.61:37771: connect: Connection refused (error 111)
W20260927 14:56:04.265519 23101 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260927 14:56:04.265843 23101 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 14:56:04.265913 23101 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 14:56:04.265985 23101 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 14:56:04.269930 23101 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 14:56:04.270007 23101 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 14:56:04.270046 23101 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 14:56:04.270095 23101 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 14:56:04.270131 23101 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 14:56:04.272327 23094 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37771 } attempt: 2
I20260927 14:56:04.273159 23098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:56:04.272796 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:37771 (local address 127.0.0.1:53932)
0927 14:56:04.272933 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0927 14:56:04.272996 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:37771: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260927 14:56:04.273823 23098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:56:04.273617 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:37771 (local address 127.0.0.1:53936)
0927 14:56:04.273719 (+ 102us) negotiation.cc:107] Waiting for socket to connect
0927 14:56:04.273765 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:37771: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260927 14:56:04.273938 23094 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:37771: Network error: Client connection negotiation failed: client connection to 127.0.98.61:37771: connect: Connection refused (error 111)
I20260927 14:56:04.275658 23101 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44519
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790520684602251-392-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790520684602251-392-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:45019,127.0.98.61:37771,127.0.98.60:37403
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790520684602251-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:37771
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790520684602251-392-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.98.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23101
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790520684602251-392-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:15:17 UTC on ccycloud.quasar-ledxfe.root.comops.site
build id 4676134
I20260927 14:56:04.276695 23101 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 14:56:04.277710 23101 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 14:56:04.285794 23106 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 14:56:04.285797 23107 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 14:56:04.285840 23109 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 14:56:04.285997 23101 server_base.cc:1034] running on GCE node
I20260927 14:56:04.286569 23101 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 14:56:04.287572 23101 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 14:56:04.288748 23101 hybrid_clock.cc:648] HybridClock initialized: now 1790520964288733 us; error 28 us; skew 500 ppm
Sep 27 14:56:04 dist-test-slave-xk1g krb5kdc[23013](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790520964, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 14:56:04.291421 23101 init.cc:377] Logged in from keytab as kudu/127.0.98.61@KRBTEST.COM (short username kudu)
I20260927 14:56:04.292270 23101 webserver.cc:460] Webserver started at http://127.0.98.61:38757/ using document root <none> and password file <none>
I20260927 14:56:04.292825 23101 fs_manager.cc:359] Metadata directory not provided
I20260927 14:56:04.292908 23101 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 14:56:04.293170 23101 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 14:56:04.295059 23101 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6936m5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790520684602251-392-0/minicluster-data/master-1/data/instance:
uuid: "b3d14199395641b5a35166c3ab10ed38"
format_stamp: "Formatted at 2026-09-27 14:56:04 on dist-test-slave-xk1g"
server_key: "f7ee3e1ba64ad30bc29a03c536e9fdb1"
server_key_iv: "cecaa54bdc6e8bc9f1cfc89199b1b82d"
server_key_version: "encryptionkey@0"
I20260927 14:56:04.295679 23101 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6936m5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790520684602251-392-0/minicluster-data/master-1/wal/instance:
uuid: "b3d14199395641b5a35166c3ab10ed38"
format_stamp: "Formatted at 2026-09-27 14:56:04 on dist-test-slave-xk1g"
server_key: "f7ee3e1ba64ad30bc29a03c536e9fdb1"
server_key_iv: "cecaa54bdc6e8bc9f1cfc89199b1b82d"
server_key_version: "encryptionkey@0"
I20260927 14:56:04.299111 23101 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260927 14:56:04.301297 23116 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 14:56:04.302386 23101 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260927 14:56:04.302517 23101 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790520684602251-392-0/minicluster-data/master-1/data,/tmp/dist-test-task6936m5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790520684602251-392-0/minicluster-data/master-1/wal
uuid: "b3d14199395641b5a35166c3ab10ed38"
format_stamp: "Formatted at 2026-09-27 14:56:04 on dist-test-slave-xk1g"
server_key: "f7ee3e1ba64ad30bc29a03c536e9fdb1"
server_key_iv: "cecaa54bdc6e8bc9f1cfc89199b1b82d"
server_key_version: "encryptionkey@0"
I20260927 14:56:04.302616 23101 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790520684602251-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790520684602251-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790520684602251-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 14:56:04.318692 23101 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 14:56:04.321874 23101 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 14:56:04.322068 23101 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 14:56:04.328728 23101 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:37771
I20260927 14:56:04.328990 23168 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:37771 every 8 connection(s)
I20260927 14:56:04.329695 23101 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6936m5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790520684602251-392-0/minicluster-data/master-1/data/info.pb
I20260927 14:56:04.332273 23169 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 14:56:04.334002 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6936m5/build/debug/bin/kudu as pid 23101
I20260927 14:56:04.334133 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6936m5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790520684602251-392-0/minicluster-data/master-1/wal/instance
I20260927 14:56:04.334414 392 external_mini_cluster.cc:1428] Setting key ddc414318c60f921e8b029ef1cc3d79b
I20260927 14:56:04.336944 23169 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45019 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.0.98.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.60@KRBTEST.COM" created.
Sep 27 14:56:04 dist-test-slave-xk1g krb5kdc[23013](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790520964, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 14:56:04.350495 23097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:56:04.338904 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54880 (local address 127.0.98.62:45019)
0927 14:56:04.339088 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0927 14:56:04.339096 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 14:56:04.339820 (+ 724us) server_negotiation.cc:407] Connection header received
0927 14:56:04.340726 (+ 906us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:56:04.340729 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:56:04.340786 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:56:04.340898 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:56:04.342050 (+ 1152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:56:04.342587 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:56:04.343792 (+ 1205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:56:04.344026 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:56:04.344104 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:56:04.346971 (+ 2867us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:56:04.346974 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:56:04.346977 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 14:56:04.347000 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:56:04.348442 (+ 1442us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:56:04.348939 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:56:04.348945 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:56:04.348946 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:56:04.349031 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:56:04.349389 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:56:04.349394 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:56:04.349396 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:56:04.349583 (+ 187us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:56:04.349685 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0927 14:56:04.350190 (+ 505us) server_negotiation.cc:299] Negotiation successful
0927 14:56:04.350320 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260927 14:56:04.351303 23172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:56:04.339214 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45019 (local address 127.0.0.1:54880)
0927 14:56:04.339658 (+ 444us) negotiation.cc:107] Waiting for socket to connect
0927 14:56:04.339709 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0927 14:56:04.340524 (+ 815us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:56:04.341036 (+ 512us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:56:04.341042 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:56:04.341426 (+ 384us) client_negotiation.cc:189] Negotiated authn=SASL
0927 14:56:04.341906 (+ 480us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:56:04.341920 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:56:04.342730 (+ 810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:56:04.342736 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:56:04.343661 (+ 925us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:56:04.343674 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:56:04.344197 (+ 523us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:56:04.344200 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:56:04.344359 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:56:04.344934 (+ 575us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 14:56:04.344961 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0927 14:56:04.346807 (+ 1846us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 14:56:04.348563 (+ 1756us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 14:56:04.348569 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 14:56:04.348583 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0927 14:56:04.348839 (+ 256us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 14:56:04.349153 (+ 314us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 14:56:04.349156 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 14:56:04.349158 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 14:56:04.349297 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 14:56:04.349685 (+ 388us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 14:56:04.349693 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 14:56:04.349951 (+ 258us) client_negotiation.cc:769] Sending connection context
0927 14:56:04.350138 (+ 187us) client_negotiation.cc:240] Negotiation successful
0927 14:56:04.350466 (+ 328us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":250,"thread_start_us":132,"threads_started":1}
I20260927 14:56:04.352771 23169 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37771 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 14:56:04 dist-test-slave-xk1g krb5kdc[23013](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790520964, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6936m5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790520684602251-392-0/krb5kdc/kudu_127.0.98.60.keytab.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6936m5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790520684602251-392-0/krb5kdc/kudu_127.0.98.60.keytab.
I20260927 14:56:04.360677 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6936m5/build/debug/bin/kudu
/tmp/dist-test-task6936m5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790520684602251-392-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790520684602251-392-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790520684602251-392-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790520684602251-392-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:37403
--webserver_interface=127.0.98.60
--webserver_port=0
--keytab_file=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790520684602251-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44519
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:45019,127.0.98.61:37771,127.0.98.60:37403
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790520684602251-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790520684602251-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790520684602251-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 14:56:04.361778 23172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:56:04.353319 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:37771 (local address 127.0.0.1:53950)
0927 14:56:04.353413 (+ 94us) negotiation.cc:107] Waiting for socket to connect
0927 14:56:04.353425 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 14:56:04.353491 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:56:04.354090 (+ 599us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:56:04.354093 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:56:04.354224 (+ 131us) client_negotiation.cc:189] Negotiated authn=SASL
0927 14:56:04.354309 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:56:04.354314 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:56:04.355269 (+ 955us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:56:04.355272 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:56:04.355845 (+ 573us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:56:04.355854 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:56:04.356429 (+ 575us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:56:04.356436 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:56:04.356552 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:56:04.357021 (+ 469us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 14:56:04.357038 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 14:56:04.358237 (+ 1199us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 14:56:04.360136 (+ 1899us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 14:56:04.360139 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 14:56:04.360141 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 14:56:04.360383 (+ 242us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 14:56:04.360660 (+ 277us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 14:56:04.360664 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 14:56:04.360667 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 14:56:04.360763 (+ 96us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 14:56:04.361347 (+ 584us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 14:56:04.361350 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 14:56:04.361433 (+ 83us) client_negotiation.cc:769] Sending connection context
0927 14:56:04.361540 (+ 107us) client_negotiation.cc:240] Negotiation successful
0927 14:56:04.361645 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31}
I20260927 14:56:04.362071 23174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:56:04.353436 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53950 (local address 127.0.98.61:37771)
0927 14:56:04.353750 (+ 314us) server_negotiation.cc:206] Beginning negotiation
0927 14:56:04.353760 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 14:56:04.353778 (+ 18us) server_negotiation.cc:407] Connection header received
0927 14:56:04.353812 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:56:04.353816 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:56:04.353961 (+ 145us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:56:04.354070 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:56:04.354398 (+ 328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:56:04.355186 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:56:04.355934 (+ 748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:56:04.356314 (+ 380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:56:04.356404 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:56:04.358360 (+ 1956us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:56:04.358369 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:56:04.358379 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 14:56:04.358399 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:56:04.360038 (+ 1639us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:56:04.360491 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:56:04.360496 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:56:04.360497 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:56:04.360573 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:56:04.360873 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:56:04.360879 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:56:04.360883 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:56:04.361220 (+ 337us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:56:04.361358 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0927 14:56:04.361640 (+ 282us) server_negotiation.cc:299] Negotiation successful
0927 14:56:04.361804 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":188,"thread_start_us":116,"threads_started":1}
I20260927 14:56:04.364190 23169 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37403 } has no permanent_uuid. Determining permanent_uuid...
I20260927 14:56:04.365036 23172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:56:04.364664 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:37403 (local address 127.0.0.1:40916)
0927 14:56:04.364842 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0927 14:56:04.364946 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37403: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260927 14:56:04.365099 23094 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37771 } attempt: 3
W20260927 14:56:04.365413 23117 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37403: connect: Connection refused (error 111)
I20260927 14:56:04.366811 23172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:56:04.366244 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:37403 (local address 127.0.0.1:40928)
0927 14:56:04.366726 (+ 482us) negotiation.cc:107] Waiting for socket to connect
0927 14:56:04.366766 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37403: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":431}
W20260927 14:56:04.367009 23169 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:37403: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37403: connect: Connection refused (error 111)
Sep 27 14:56:04 dist-test-slave-xk1g krb5kdc[23013](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790520964, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260927 14:56:04.374522 23098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:56:04.365573 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:37771 (local address 127.0.0.1:53952)
0927 14:56:04.365720 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0927 14:56:04.365735 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 14:56:04.365815 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:56:04.366348 (+ 533us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:56:04.366351 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:56:04.366502 (+ 151us) client_negotiation.cc:189] Negotiated authn=SASL
0927 14:56:04.366593 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:56:04.366598 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:56:04.367689 (+ 1091us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:56:04.367692 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:56:04.368607 (+ 915us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:56:04.368616 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:56:04.369362 (+ 746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:56:04.369364 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:56:04.369443 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:56:04.369888 (+ 445us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 14:56:04.369900 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 14:56:04.371148 (+ 1248us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 14:56:04.373219 (+ 2071us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 14:56:04.373222 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 14:56:04.373223 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 14:56:04.373407 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 14:56:04.373683 (+ 276us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 14:56:04.373685 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 14:56:04.373687 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 14:56:04.373754 (+ 67us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 14:56:04.374153 (+ 399us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 14:56:04.374155 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 14:56:04.374231 (+ 76us) client_negotiation.cc:769] Sending connection context
0927 14:56:04.374308 (+ 77us) client_negotiation.cc:240] Negotiation successful
0927 14:56:04.374405 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":54}
I20260927 14:56:04.374833 23174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:56:04.365926 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53952 (local address 127.0.98.61:37771)
0927 14:56:04.366071 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0927 14:56:04.366079 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 14:56:04.366098 (+ 19us) server_negotiation.cc:407] Connection header received
0927 14:56:04.366143 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:56:04.366148 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:56:04.366205 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:56:04.366299 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:56:04.366706 (+ 407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:56:04.367563 (+ 857us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:56:04.368728 (+ 1165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:56:04.369245 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:56:04.369381 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:56:04.371282 (+ 1901us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:56:04.371290 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:56:04.371294 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 14:56:04.371326 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:56:04.373115 (+ 1789us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:56:04.373497 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:56:04.373501 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:56:04.373503 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:56:04.373596 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:56:04.373838 (+ 242us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:56:04.373842 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:56:04.373844 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:56:04.374062 (+ 218us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:56:04.374168 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0927 14:56:04.374388 (+ 220us) server_negotiation.cc:299] Negotiation successful
0927 14:56:04.374508 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260927 14:56:04.375581 23094 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37403 } has no permanent_uuid. Determining permanent_uuid...
I20260927 14:56:04.376216 23098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:56:04.376012 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:37403 (local address 127.0.0.1:40936)
0927 14:56:04.376116 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0927 14:56:04.376160 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37403: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20260927 14:56:04.376833 23098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:56:04.376622 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:37403 (local address 127.0.0.1:40942)
0927 14:56:04.376742 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0927 14:56:04.376790 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37403: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
W20260927 14:56:04.376966 23094 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:37403: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37403: connect: Connection refused (error 111)
I20260927 14:56:04.408102 23094 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37403 } attempt: 1
I20260927 14:56:04.408907 23098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:56:04.408631 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:37403 (local address 127.0.0.1:40950)
0927 14:56:04.408773 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0927 14:56:04.408833 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37403: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260927 14:56:04.409638 23098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:56:04.409422 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:37403 (local address 127.0.0.1:40966)
0927 14:56:04.409542 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0927 14:56:04.409589 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37403: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260927 14:56:04.409752 23094 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:37403: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37403: connect: Connection refused (error 111)
I20260927 14:56:04.416249 23169 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37403 } attempt: 1
I20260927 14:56:04.416862 23172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:56:04.416625 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:37403 (local address 127.0.0.1:40976)
0927 14:56:04.416742 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0927 14:56:04.416798 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37403: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260927 14:56:04.417496 23172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:56:04.417302 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:37403 (local address 127.0.0.1:40990)
0927 14:56:04.417398 (+ 96us) negotiation.cc:107] Waiting for socket to connect
0927 14:56:04.417444 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37403: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260927 14:56:04.417598 23169 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:37403: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37403: connect: Connection refused (error 111)
W20260927 14:56:04.474924 23175 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260927 14:56:04.475268 23175 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 14:56:04.475332 23175 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 14:56:04.475414 23175 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 14:56:04.479521 23175 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 14:56:04.479607 23175 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 14:56:04.479650 23175 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 14:56:04.479708 23175 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 14:56:04.479753 23175 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 14:56:04.484915 23094 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37403 } attempt: 2
I20260927 14:56:04.484997 23175 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44519
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790520684602251-392-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790520684602251-392-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:45019,127.0.98.61:37771,127.0.98.60:37403
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790520684602251-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:37403
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790520684602251-392-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.98.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23175
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6936m5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790520684602251-392-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:15:17 UTC on ccycloud.quasar-ledxfe.root.comops.site
build id 4676134
I20260927 14:56:04.485723 23098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:56:04.485442 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:37403 (local address 127.0.0.1:41000)
0927 14:56:04.485573 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0927 14:56:04.485635 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37403: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260927 14:56:04.485750 23169 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37403 } attempt: 2
I20260927 14:56:04.486222 23175 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 14:56:04.486444 23172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:56:04.486166 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:37403 (local address 127.0.0.1:41008)
0927 14:56:04.486305 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0927 14:56:04.486369 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37403: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260927 14:56:04.486521 23098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:56:04.486235 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:37403 (local address 127.0.0.1:41018)
0927 14:56:04.486390 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0927 14:56:04.486450 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37403: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
W20260927 14:56:04.486640 23094 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:37403: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37403: connect: Connection refused (error 111)
I20260927 14:56:04.487218 23172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:56:04.487033 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:37403 (local address 127.0.0.1:41026)
0927 14:56:04.487130 (+ 97us) negotiation.cc:107] Waiting for socket to connect
0927 14:56:04.487170 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37403: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260927 14:56:04.487353 23169 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:37403: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37403: connect: Connection refused (error 111)
I20260927 14:56:04.487481 23175 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 14:56:04.495841 23184 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 14:56:04.496052 23181 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 14:56:04.496119 23182 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 14:56:04.496230 23175 server_base.cc:1034] running on GCE node
I20260927 14:56:04.496778 23175 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 14:56:04.497933 23175 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 14:56:04.499090 23175 hybrid_clock.cc:648] HybridClock initialized: now 1790520964499075 us; error 30 us; skew 500 ppm
Sep 27 14:56:04 dist-test-slave-xk1g krb5kdc[23013](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790520964, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 14:56:04.501753 23175 init.cc:377] Logged in from keytab as kudu/127.0.98.60@KRBTEST.COM (short username kudu)
I20260927 14:56:04.502547 23175 webserver.cc:460] Webserver started at http://127.0.98.60:38995/ using document root <none> and password file <none>
I20260927 14:56:04.503067 23175 fs_manager.cc:359] Metadata directory not provided
I20260927 14:56:04.503149 23175 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 14:56:04.503358 23175 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 14:56:04.505398 23175 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6936m5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790520684602251-392-0/minicluster-data/master-2/data/instance:
uuid: "157c37291792449187ddf3ec01619734"
format_stamp: "Formatted at 2026-09-27 14:56:04 on dist-test-slave-xk1g"
server_key: "e0eaa85cceb2f1656e2f3910999d1ad9"
server_key_iv: "d4503a3ad86643e1507f8810c0ded90a"
server_key_version: "encryptionkey@0"
I20260927 14:56:04.505975 23175 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6936m5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790520684602251-392-0/minicluster-data/master-2/wal/instance:
uuid: "157c37291792449187ddf3ec01619734"
format_stamp: "Formatted at 2026-09-27 14:56:04 on dist-test-slave-xk1g"
server_key: "e0eaa85cceb2f1656e2f3910999d1ad9"
server_key_iv: "d4503a3ad86643e1507f8810c0ded90a"
server_key_version: "encryptionkey@0"
I20260927 14:56:04.509475 23175 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260927 14:56:04.511745 23191 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 14:56:04.512881 23175 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 14:56:04.513015 23175 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790520684602251-392-0/minicluster-data/master-2/data,/tmp/dist-test-task6936m5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790520684602251-392-0/minicluster-data/master-2/wal
uuid: "157c37291792449187ddf3ec01619734"
format_stamp: "Formatted at 2026-09-27 14:56:04 on dist-test-slave-xk1g"
server_key: "e0eaa85cceb2f1656e2f3910999d1ad9"
server_key_iv: "d4503a3ad86643e1507f8810c0ded90a"
server_key_version: "encryptionkey@0"
I20260927 14:56:04.513157 23175 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790520684602251-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790520684602251-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task6936m5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790520684602251-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 14:56:04.527227 23175 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 14:56:04.530413 23175 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 14:56:04.530618 23175 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 14:56:04.537292 23175 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:37403
I20260927 14:56:04.537571 23243 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:37403 every 8 connection(s)
I20260927 14:56:04.538271 23175 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6936m5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790520684602251-392-0/minicluster-data/master-2/data/info.pb
I20260927 14:56:04.540874 23244 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 14:56:04.544390 23244 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45019 } has no permanent_uuid. Determining permanent_uuid...
I20260927 14:56:04.547874 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6936m5/build/debug/bin/kudu as pid 23175
I20260927 14:56:04.548004 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6936m5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790520684602251-392-0/minicluster-data/master-2/wal/instance
I20260927 14:56:04.548300 392 external_mini_cluster.cc:1428] Setting key cac08276e498db4f4405133ab3b730f3
Sep 27 14:56:04 dist-test-slave-xk1g krb5kdc[23013](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790520964, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
Sep 27 14:56:04 dist-test-slave-xk1g krb5kdc[23013](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790520963, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260927 14:56:04.558133 23097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:56:04.546156 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54892 (local address 127.0.98.62:45019)
0927 14:56:04.546310 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0927 14:56:04.546314 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 14:56:04.547070 (+ 756us) server_negotiation.cc:407] Connection header received
0927 14:56:04.547916 (+ 846us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:56:04.547922 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:56:04.547990 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:56:04.548079 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:56:04.549561 (+ 1482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:56:04.550086 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:56:04.551606 (+ 1520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:56:04.551825 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:56:04.551931 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:56:04.554550 (+ 2619us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:56:04.554555 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:56:04.554558 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 14:56:04.554587 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:56:04.556003 (+ 1416us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:56:04.556573 (+ 570us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:56:04.556576 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:56:04.556577 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:56:04.556677 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:56:04.557017 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:56:04.557019 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:56:04.557020 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:56:04.557309 (+ 289us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:56:04.557414 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0927 14:56:04.557863 (+ 449us) server_negotiation.cc:299] Negotiation successful
0927 14:56:04.557997 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260927 14:56:04.559058 23246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:56:04.546511 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45019 (local address 127.0.0.1:54892)
0927 14:56:04.546923 (+ 412us) negotiation.cc:107] Waiting for socket to connect
0927 14:56:04.546969 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0927 14:56:04.547714 (+ 745us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:56:04.548257 (+ 543us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:56:04.548263 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:56:04.548677 (+ 414us) client_negotiation.cc:189] Negotiated authn=SASL
0927 14:56:04.549408 (+ 731us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:56:04.549427 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:56:04.550186 (+ 759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:56:04.550188 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:56:04.551481 (+ 1293us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:56:04.551489 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:56:04.551974 (+ 485us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:56:04.551978 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:56:04.552180 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:56:04.552737 (+ 557us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 14:56:04.552756 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0927 14:56:04.554403 (+ 1647us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 14:56:04.556108 (+ 1705us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 14:56:04.556115 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 14:56:04.556122 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0927 14:56:04.556483 (+ 361us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 14:56:04.556772 (+ 289us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 14:56:04.556774 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 14:56:04.556776 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 14:56:04.556920 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 14:56:04.557412 (+ 492us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 14:56:04.557416 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 14:56:04.557633 (+ 217us) client_negotiation.cc:769] Sending connection context
0927 14:56:04.557784 (+ 151us) client_negotiation.cc:240] Negotiation successful
0927 14:56:04.558061 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":231,"mutex_wait_us":44,"thread_start_us":106,"threads_started":1}
I20260927 14:56:04.559896 23247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:56:04.548905 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41040 (local address 127.0.98.60:37403)
0927 14:56:04.549196 (+ 291us) server_negotiation.cc:206] Beginning negotiation
0927 14:56:04.549205 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 14:56:04.549413 (+ 208us) server_negotiation.cc:407] Connection header received
0927 14:56:04.550204 (+ 791us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:56:04.550208 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:56:04.550360 (+ 152us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:56:04.550477 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:56:04.551308 (+ 831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:56:04.552041 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:56:04.553122 (+ 1081us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:56:04.553487 (+ 365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:56:04.553612 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:56:04.556053 (+ 2441us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:56:04.556062 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:56:04.556074 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 14:56:04.556099 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:56:04.557891 (+ 1792us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:56:04.558346 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:56:04.558351 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:56:04.558353 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:56:04.558433 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:56:04.558680 (+ 247us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:56:04.558683 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:56:04.558685 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:56:04.558930 (+ 245us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:56:04.559110 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0927 14:56:04.559602 (+ 492us) server_negotiation.cc:299] Negotiation successful
0927 14:56:04.559742 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":200,"thread_start_us":99,"threads_started":1}
I20260927 14:56:04.560369 23244 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37771 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 14:56:04 dist-test-slave-xk1g krb5kdc[23013](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790520964, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
Sep 27 14:56:04 dist-test-slave-xk1g krb5kdc[23013](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790520963, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 14:56:04.568631 23174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:56:04.561076 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53960 (local address 127.0.98.61:37771)
0927 14:56:04.561205 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0927 14:56:04.561209 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 14:56:04.561223 (+ 14us) server_negotiation.cc:407] Connection header received
0927 14:56:04.561266 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:56:04.561268 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:56:04.561304 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:56:04.561386 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:56:04.561821 (+ 435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:56:04.562400 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:56:04.563135 (+ 735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:56:04.563357 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:56:04.563463 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:56:04.565331 (+ 1868us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:56:04.565334 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:56:04.565335 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 14:56:04.565358 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:56:04.566918 (+ 1560us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:56:04.567381 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:56:04.567385 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:56:04.567388 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:56:04.567482 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:56:04.567787 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:56:04.567790 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:56:04.567791 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:56:04.568040 (+ 249us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:56:04.568137 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0927 14:56:04.568428 (+ 291us) server_negotiation.cc:299] Negotiation successful
0927 14:56:04.568519 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260927 14:56:04.568689 23246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:56:04.560894 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:37771 (local address 127.0.0.1:53960)
0927 14:56:04.561016 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0927 14:56:04.561031 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 14:56:04.561117 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:56:04.561397 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:56:04.561400 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:56:04.561577 (+ 177us) client_negotiation.cc:189] Negotiated authn=SASL
0927 14:56:04.561675 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:56:04.561682 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:56:04.562512 (+ 830us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:56:04.562518 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:56:04.563024 (+ 506us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:56:04.563031 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:56:04.563464 (+ 433us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:56:04.563468 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:56:04.563560 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:56:04.563982 (+ 422us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 14:56:04.563999 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 14:56:04.565202 (+ 1203us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 14:56:04.567034 (+ 1832us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 14:56:04.567037 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 14:56:04.567039 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 14:56:04.567283 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 14:56:04.567596 (+ 313us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 14:56:04.567601 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 14:56:04.567603 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 14:56:04.567694 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 14:56:04.568139 (+ 445us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 14:56:04.568142 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 14:56:04.568256 (+ 114us) client_negotiation.cc:769] Sending connection context
0927 14:56:04.568368 (+ 112us) client_negotiation.cc:240] Negotiation successful
0927 14:56:04.568501 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":60}
I20260927 14:56:04.569350 23244 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37403 } has no permanent_uuid. Determining permanent_uuid...
I20260927 14:56:04.570608 23097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:56:04.562044 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54900 (local address 127.0.98.62:45019)
0927 14:56:04.562157 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0927 14:56:04.562161 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 14:56:04.562174 (+ 13us) server_negotiation.cc:407] Connection header received
0927 14:56:04.562255 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:56:04.562258 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:56:04.562294 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:56:04.562377 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:56:04.562920 (+ 543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:56:04.563385 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:56:04.564104 (+ 719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:56:04.564326 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:56:04.564442 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:56:04.567083 (+ 2641us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:56:04.567088 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:56:04.567092 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 14:56:04.567117 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:56:04.568901 (+ 1784us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:56:04.569387 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:56:04.569390 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:56:04.569391 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:56:04.569486 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:56:04.569751 (+ 265us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:56:04.569754 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:56:04.569756 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:56:04.569979 (+ 223us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:56:04.570207 (+ 228us) server_negotiation.cc:1035] Waiting for connection context
0927 14:56:04.570328 (+ 121us) server_negotiation.cc:299] Negotiation successful
0927 14:56:04.570439 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
Sep 27 14:56:04 dist-test-slave-xk1g krb5kdc[23013](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790520964, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
Sep 27 14:56:04 dist-test-slave-xk1g krb5kdc[23013](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790520963, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260927 14:56:04.577123 23246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:56:04.569809 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:37403 (local address 127.0.0.1:41056)
0927 14:56:04.569899 (+ 90us) negotiation.cc:107] Waiting for socket to connect
0927 14:56:04.569915 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 14:56:04.569980 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:56:04.570175 (+ 195us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:56:04.570178 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:56:04.570321 (+ 143us) client_negotiation.cc:189] Negotiated authn=SASL
0927 14:56:04.570373 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:56:04.570377 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:56:04.571190 (+ 813us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:56:04.571196 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:56:04.571858 (+ 662us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:56:04.571870 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:56:04.572323 (+ 453us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:56:04.572326 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:56:04.572397 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:56:04.572821 (+ 424us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 14:56:04.572833 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 14:56:04.574124 (+ 1291us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 14:56:04.575601 (+ 1477us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 14:56:04.575604 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 14:56:04.575606 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 14:56:04.575858 (+ 252us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 14:56:04.576131 (+ 273us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 14:56:04.576133 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 14:56:04.576134 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 14:56:04.576204 (+ 70us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 14:56:04.576603 (+ 399us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 14:56:04.576608 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 14:56:04.576710 (+ 102us) client_negotiation.cc:769] Sending connection context
0927 14:56:04.576828 (+ 118us) client_negotiation.cc:240] Negotiation successful
0927 14:56:04.576959 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":27}
I20260927 14:56:04.577251 23247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:56:04.569861 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41056 (local address 127.0.98.60:37403)
0927 14:56:04.569970 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0927 14:56:04.569974 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 14:56:04.569985 (+ 11us) server_negotiation.cc:407] Connection header received
0927 14:56:04.570051 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:56:04.570055 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:56:04.570091 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:56:04.570165 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:56:04.570468 (+ 303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:56:04.571107 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:56:04.571972 (+ 865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:56:04.572223 (+ 251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:56:04.572323 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:56:04.574237 (+ 1914us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:56:04.574244 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:56:04.574249 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 14:56:04.574279 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:56:04.575501 (+ 1222us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:56:04.575951 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:56:04.575955 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:56:04.575957 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:56:04.576034 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:56:04.576282 (+ 248us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:56:04.576285 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:56:04.576287 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:56:04.576508 (+ 221us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:56:04.576584 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0927 14:56:04.576907 (+ 323us) server_negotiation.cc:299] Negotiation successful
0927 14:56:04.577021 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260927 14:56:04.578536 23169 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37403 } attempt: 3
I20260927 14:56:04.580274 23174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:56:04.571617 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53970 (local address 127.0.98.61:37771)
0927 14:56:04.571709 (+ 92us) server_negotiation.cc:206] Beginning negotiation
0927 14:56:04.571712 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 14:56:04.571724 (+ 12us) server_negotiation.cc:407] Connection header received
0927 14:56:04.571784 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:56:04.571786 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:56:04.571824 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:56:04.571909 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:56:04.572404 (+ 495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:56:04.572869 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:56:04.573987 (+ 1118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:56:04.574208 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:56:04.574314 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:56:04.576915 (+ 2601us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:56:04.576918 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:56:04.576921 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 14:56:04.576942 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:56:04.578640 (+ 1698us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:56:04.579127 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:56:04.579130 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:56:04.579131 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:56:04.579211 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:56:04.579483 (+ 272us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:56:04.579489 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:56:04.579491 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:56:04.579728 (+ 237us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:56:04.579966 (+ 238us) server_negotiation.cc:1035] Waiting for connection context
0927 14:56:04.580039 (+ 73us) server_negotiation.cc:299] Negotiation successful
0927 14:56:04.580140 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":22,"server-negotiator.queue_time_us":37}
I20260927 14:56:04.581154 23244 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 157c37291792449187ddf3ec01619734: Bootstrap starting.
Sep 27 14:56:04 dist-test-slave-xk1g krb5kdc[23013](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790520964, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260927 14:56:04.583822 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 14:56:04.584038 23244 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 157c37291792449187ddf3ec01619734: Neither blocks nor log segments found. Creating new log.
I20260927 14:56:04.585798 23094 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37403 } attempt: 3
I20260927 14:56:04.585103 23068 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "ebd09ac4557a413dba8bcbfb01fcc73e"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54900
I20260927 14:56:04.586628 23244 log.cc:824] T 00000000000000000000000000000000 P 157c37291792449187ddf3ec01619734: Log is configured to *not* fsync() on all Append() calls
I20260927 14:56:04.587298 23172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:56:04.578893 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:37403 (local address 127.0.0.1:41070)
0927 14:56:04.579071 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0927 14:56:04.579086 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 14:56:04.579149 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:56:04.579345 (+ 196us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:56:04.579348 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:56:04.579517 (+ 169us) client_negotiation.cc:189] Negotiated authn=SASL
0927 14:56:04.579622 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:56:04.579627 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:56:04.580513 (+ 886us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:56:04.580520 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:56:04.581416 (+ 896us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:56:04.581428 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:56:04.582020 (+ 592us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:56:04.582024 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:56:04.582104 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:56:04.582546 (+ 442us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 14:56:04.582560 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0927 14:56:04.583946 (+ 1386us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 14:56:04.585668 (+ 1722us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 14:56:04.585673 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 14:56:04.585675 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 14:56:04.585878 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 14:56:04.586230 (+ 352us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 14:56:04.586233 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 14:56:04.586235 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 14:56:04.586327 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 14:56:04.586809 (+ 482us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 14:56:04.586812 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 14:56:04.586920 (+ 108us) client_negotiation.cc:769] Sending connection context
0927 14:56:04.587003 (+ 83us) client_negotiation.cc:240] Negotiation successful
0927 14:56:04.587129 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":102,"mutex_wait_us":2}
I20260927 14:56:04.587922 23247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:56:04.578996 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41070 (local address 127.0.98.60:37403)
0927 14:56:04.579112 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0927 14:56:04.579118 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 14:56:04.579134 (+ 16us) server_negotiation.cc:407] Connection header received
0927 14:56:04.579222 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:56:04.579225 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:56:04.579259 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:56:04.579333 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:56:04.579725 (+ 392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:56:04.580397 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:56:04.581566 (+ 1169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:56:04.581875 (+ 309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:56:04.582012 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:56:04.584064 (+ 2052us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:56:04.584068 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:56:04.584071 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 14:56:04.584095 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:56:04.585558 (+ 1463us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:56:04.586011 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:56:04.586014 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:56:04.586016 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:56:04.586119 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:56:04.586442 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:56:04.586446 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:56:04.586449 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:56:04.586691 (+ 242us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:56:04.587571 (+ 880us) server_negotiation.cc:1035] Waiting for connection context
0927 14:56:04.587645 (+ 74us) server_negotiation.cc:299] Negotiation successful
0927 14:56:04.587756 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260927 14:56:04.589501 23244 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 157c37291792449187ddf3ec01619734: No bootstrap required, opened a new log
Sep 27 14:56:04 dist-test-slave-xk1g krb5kdc[23013](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790520964, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260927 14:56:04.590869 23169 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b3d14199395641b5a35166c3ab10ed38: Bootstrap starting.
I20260927 14:56:04.593554 23169 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b3d14199395641b5a35166c3ab10ed38: Neither blocks nor log segments found. Creating new log.
I20260927 14:56:04.593538 23244 raft_consensus.cc:348] T 00000000000000000000000000000000 P 157c37291792449187ddf3ec01619734 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebd09ac4557a413dba8bcbfb01fcc73e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45019 } } peers { permanent_uuid: "b3d14199395641b5a35166c3ab10ed38" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37771 } } peers { permanent_uuid: "157c37291792449187ddf3ec01619734" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37403 } }
I20260927 14:56:04.593780 23244 raft_consensus.cc:729] T 00000000000000000000000000000000 P 157c37291792449187ddf3ec01619734 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 157c37291792449187ddf3ec01619734, State: Initialized, Role: FOLLOWER
I20260927 14:56:04.594122 23244 consensus_queue.cc:260] T 00000000000000000000000000000000 P 157c37291792449187ddf3ec01619734 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebd09ac4557a413dba8bcbfb01fcc73e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45019 } } peers { permanent_uuid: "b3d14199395641b5a35166c3ab10ed38" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37771 } } peers { permanent_uuid: "157c37291792449187ddf3ec01619734" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37403 } }
I20260927 14:56:04.594383 23169 log.cc:824] T 00000000000000000000000000000000 P b3d14199395641b5a35166c3ab10ed38: Log is configured to *not* fsync() on all Append() calls
I20260927 14:56:04.594890 23098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:56:04.586157 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:37403 (local address 127.0.0.1:41078)
0927 14:56:04.586283 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0927 14:56:04.586298 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 14:56:04.586383 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:56:04.586921 (+ 538us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:56:04.586924 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:56:04.587113 (+ 189us) client_negotiation.cc:189] Negotiated authn=SASL
0927 14:56:04.587217 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:56:04.587224 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:56:04.587995 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:56:04.587999 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:56:04.588609 (+ 610us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:56:04.588618 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:56:04.589245 (+ 627us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:56:04.589249 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:56:04.589334 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:56:04.589822 (+ 488us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 14:56:04.589842 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0927 14:56:04.591171 (+ 1329us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 14:56:04.593305 (+ 2134us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 14:56:04.593308 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 14:56:04.593310 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 14:56:04.593509 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 14:56:04.593788 (+ 279us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 14:56:04.593790 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 14:56:04.593793 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 14:56:04.593897 (+ 104us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 14:56:04.594340 (+ 443us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 14:56:04.594345 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 14:56:04.594447 (+ 102us) client_negotiation.cc:769] Sending connection context
0927 14:56:04.594574 (+ 127us) client_negotiation.cc:240] Negotiation successful
0927 14:56:04.594717 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":36}
I20260927 14:56:04.594753 23251 sys_catalog.cc:455] T 00000000000000000000000000000000 P 157c37291792449187ddf3ec01619734 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebd09ac4557a413dba8bcbfb01fcc73e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45019 } } peers { permanent_uuid: "b3d14199395641b5a35166c3ab10ed38" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37771 } } peers { permanent_uuid: "157c37291792449187ddf3ec01619734" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37403 } } }
I20260927 14:56:04.595306 23251 sys_catalog.cc:458] T 00000000000000000000000000000000 P 157c37291792449187ddf3ec01619734 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 14:56:04.595561 23244 sys_catalog.cc:564] T 00000000000000000000000000000000 P 157c37291792449187ddf3ec01619734 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 14:56:04.595584 23249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:56:04.586386 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41078 (local address 127.0.98.60:37403)
0927 14:56:04.586705 (+ 319us) server_negotiation.cc:206] Beginning negotiation
0927 14:56:04.586715 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 14:56:04.586731 (+ 16us) server_negotiation.cc:407] Connection header received
0927 14:56:04.586765 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:56:04.586768 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:56:04.586811 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:56:04.586898 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:56:04.587344 (+ 446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:56:04.587895 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:56:04.588720 (+ 825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:56:04.589142 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:56:04.589237 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:56:04.591308 (+ 2071us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:56:04.591311 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:56:04.591314 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 14:56:04.591336 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:56:04.593203 (+ 1867us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:56:04.593609 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:56:04.593615 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:56:04.593617 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:56:04.593704 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:56:04.593998 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:56:04.594002 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:56:04.594004 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:56:04.594210 (+ 206us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:56:04.594297 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0927 14:56:04.595103 (+ 806us) server_negotiation.cc:299] Negotiation successful
0927 14:56:04.595235 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":254,"thread_start_us":123,"threads_started":1}
I20260927 14:56:04.596782 23169 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b3d14199395641b5a35166c3ab10ed38: No bootstrap required, opened a new log
I20260927 14:56:04.600317 23169 raft_consensus.cc:348] T 00000000000000000000000000000000 P b3d14199395641b5a35166c3ab10ed38 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebd09ac4557a413dba8bcbfb01fcc73e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45019 } } peers { permanent_uuid: "b3d14199395641b5a35166c3ab10ed38" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37771 } } peers { permanent_uuid: "157c37291792449187ddf3ec01619734" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37403 } }
I20260927 14:56:04.600559 23169 raft_consensus.cc:729] T 00000000000000000000000000000000 P b3d14199395641b5a35166c3ab10ed38 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b3d14199395641b5a35166c3ab10ed38, State: Initialized, Role: FOLLOWER
I20260927 14:56:04.600896 23169 consensus_queue.cc:260] T 00000000000000000000000000000000 P b3d14199395641b5a35166c3ab10ed38 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebd09ac4557a413dba8bcbfb01fcc73e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45019 } } peers { permanent_uuid: "b3d14199395641b5a35166c3ab10ed38" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37771 } } peers { permanent_uuid: "157c37291792449187ddf3ec01619734" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37403 } }
W20260927 14:56:04.601245 23263 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 157c37291792449187ddf3ec01619734: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 14:56:04.601323 23263 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 14:56:04.603955 23094 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ebd09ac4557a413dba8bcbfb01fcc73e: Bootstrap starting.
I20260927 14:56:04.603009 23175 master_runner.cc:186] Error getting master registration for 127.0.98.62:45019: OK, instance_id { permanent_uuid: "ebd09ac4557a413dba8bcbfb01fcc73e" instance_seqno: 1790520964124278 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260927 14:56:04.606442 23175 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260927 14:56:04.606657 23094 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ebd09ac4557a413dba8bcbfb01fcc73e: Neither blocks nor log segments found. Creating new log.
I20260927 14:56:04.607501 23094 log.cc:824] T 00000000000000000000000000000000 P ebd09ac4557a413dba8bcbfb01fcc73e: Log is configured to *not* fsync() on all Append() calls
I20260927 14:56:04.607596 23169 sys_catalog.cc:564] T 00000000000000000000000000000000 P b3d14199395641b5a35166c3ab10ed38 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 14:56:04.609596 23094 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ebd09ac4557a413dba8bcbfb01fcc73e: No bootstrap required, opened a new log
I20260927 14:56:04.610323 23265 sys_catalog.cc:455] T 00000000000000000000000000000000 P b3d14199395641b5a35166c3ab10ed38 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebd09ac4557a413dba8bcbfb01fcc73e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45019 } } peers { permanent_uuid: "b3d14199395641b5a35166c3ab10ed38" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37771 } } peers { permanent_uuid: "157c37291792449187ddf3ec01619734" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37403 } } }
I20260927 14:56:04.610472 23265 sys_catalog.cc:458] T 00000000000000000000000000000000 P b3d14199395641b5a35166c3ab10ed38 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 14:56:04.612622 23094 raft_consensus.cc:348] T 00000000000000000000000000000000 P ebd09ac4557a413dba8bcbfb01fcc73e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebd09ac4557a413dba8bcbfb01fcc73e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45019 } } peers { permanent_uuid: "b3d14199395641b5a35166c3ab10ed38" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37771 } } peers { permanent_uuid: "157c37291792449187ddf3ec01619734" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37403 } }
I20260927 14:56:04.612839 23094 raft_consensus.cc:729] T 00000000000000000000000000000000 P ebd09ac4557a413dba8bcbfb01fcc73e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ebd09ac4557a413dba8bcbfb01fcc73e, State: Initialized, Role: FOLLOWER
I20260927 14:56:04.613189 23094 consensus_queue.cc:260] T 00000000000000000000000000000000 P ebd09ac4557a413dba8bcbfb01fcc73e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebd09ac4557a413dba8bcbfb01fcc73e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45019 } } peers { permanent_uuid: "b3d14199395641b5a35166c3ab10ed38" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37771 } } peers { permanent_uuid: "157c37291792449187ddf3ec01619734" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37403 } }
I20260927 14:56:04.614454 23094 sys_catalog.cc:564] T 00000000000000000000000000000000 P ebd09ac4557a413dba8bcbfb01fcc73e [sys.catalog]: configured and running, proceeding with master startup.
I20260927 14:56:04.614513 23068 raft_consensus.cc:482] T 00000000000000000000000000000000 P ebd09ac4557a413dba8bcbfb01fcc73e [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260927 14:56:04.614605 23068 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ebd09ac4557a413dba8bcbfb01fcc73e [term 0 FOLLOWER]: Advancing to term 1
I20260927 14:56:04.611158 23101 master_runner.cc:186] Error getting master registration for 127.0.98.62:45019: OK, instance_id { permanent_uuid: "ebd09ac4557a413dba8bcbfb01fcc73e" instance_seqno: 1790520964124278 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260927 14:56:04.615052 23101 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260927 14:56:04.615928 23068 raft_consensus.cc:504] T 00000000000000000000000000000000 P ebd09ac4557a413dba8bcbfb01fcc73e [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebd09ac4557a413dba8bcbfb01fcc73e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45019 } } peers { permanent_uuid: "b3d14199395641b5a35166c3ab10ed38" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37771 } } peers { permanent_uuid: "157c37291792449187ddf3ec01619734" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37403 } }
I20260927 14:56:04.616777 23279 sys_catalog.cc:455] T 00000000000000000000000000000000 P ebd09ac4557a413dba8bcbfb01fcc73e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebd09ac4557a413dba8bcbfb01fcc73e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45019 } } peers { permanent_uuid: "b3d14199395641b5a35166c3ab10ed38" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37771 } } peers { permanent_uuid: "157c37291792449187ddf3ec01619734" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37403 } } }
I20260927 14:56:04.616902 23279 sys_catalog.cc:458] T 00000000000000000000000000000000 P ebd09ac4557a413dba8bcbfb01fcc73e [sys.catalog]: This master's current role is: FOLLOWER
I20260927 14:56:04.617547 23143 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ebd09ac4557a413dba8bcbfb01fcc73e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "b3d14199395641b5a35166c3ab10ed38"
I20260927 14:56:04.617748 23143 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b3d14199395641b5a35166c3ab10ed38 [term 0 FOLLOWER]: Advancing to term 1
I20260927 14:56:04.618970 23143 raft_consensus.cc:2454] T 00000000000000000000000000000000 P b3d14199395641b5a35166c3ab10ed38 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ebd09ac4557a413dba8bcbfb01fcc73e in term 1.
I20260927 14:56:04.619750 23042 leader_election.cc:302] T 00000000000000000000000000000000 P ebd09ac4557a413dba8bcbfb01fcc73e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b3d14199395641b5a35166c3ab10ed38, ebd09ac4557a413dba8bcbfb01fcc73e; no voters:
I20260927 14:56:04.620011 23279 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ebd09ac4557a413dba8bcbfb01fcc73e [term 1 FOLLOWER]: Leader election won for term 1
I20260927 14:56:04.620204 23279 raft_consensus.cc:686] T 00000000000000000000000000000000 P ebd09ac4557a413dba8bcbfb01fcc73e [term 1 LEADER]: Becoming Leader. State: Replica: ebd09ac4557a413dba8bcbfb01fcc73e, State: Running, Role: LEADER
I20260927 14:56:04.620503 23279 consensus_queue.cc:237] T 00000000000000000000000000000000 P ebd09ac4557a413dba8bcbfb01fcc73e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebd09ac4557a413dba8bcbfb01fcc73e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45019 } } peers { permanent_uuid: "b3d14199395641b5a35166c3ab10ed38" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37771 } } peers { permanent_uuid: "157c37291792449187ddf3ec01619734" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37403 } }
I20260927 14:56:04.620628 23218 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ebd09ac4557a413dba8bcbfb01fcc73e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "157c37291792449187ddf3ec01619734"
W20260927 14:56:04.620822 23047 tablet.cc:2369] T 00000000000000000000000000000000 P ebd09ac4557a413dba8bcbfb01fcc73e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 14:56:04.620863 23218 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 157c37291792449187ddf3ec01619734 [term 0 FOLLOWER]: Advancing to term 1
I20260927 14:56:04.622169 23218 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 157c37291792449187ddf3ec01619734 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ebd09ac4557a413dba8bcbfb01fcc73e in term 1.
I20260927 14:56:04.622148 23279 sys_catalog.cc:455] T 00000000000000000000000000000000 P ebd09ac4557a413dba8bcbfb01fcc73e [sys.catalog]: SysCatalogTable state changed. Reason: New leader ebd09ac4557a413dba8bcbfb01fcc73e. Latest consensus state: current_term: 1 leader_uuid: "ebd09ac4557a413dba8bcbfb01fcc73e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebd09ac4557a413dba8bcbfb01fcc73e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45019 } } peers { permanent_uuid: "b3d14199395641b5a35166c3ab10ed38" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37771 } } peers { permanent_uuid: "157c37291792449187ddf3ec01619734" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37403 } } }
I20260927 14:56:04.622238 23279 sys_catalog.cc:458] T 00000000000000000000000000000000 P ebd09ac4557a413dba8bcbfb01fcc73e [sys.catalog]: This master's current role is: LEADER
I20260927 14:56:04.616820 23068 leader_election.cc:288] T 00000000000000000000000000000000 P ebd09ac4557a413dba8bcbfb01fcc73e [CANDIDATE]: Term 1 election: Requested vote from peers b3d14199395641b5a35166c3ab10ed38 (127.0.98.61:37771), 157c37291792449187ddf3ec01619734 (127.0.98.60:37403)
W20260927 14:56:04.624169 23277 catalog_manager.cc:1558] T 00000000000000000000000000000000 P b3d14199395641b5a35166c3ab10ed38: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 14:56:04.624300 23277 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20260927 14:56:04.628598 23290 catalog_manager.cc:1558] T 00000000000000000000000000000000 P ebd09ac4557a413dba8bcbfb01fcc73e: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 14:56:04.628746 23290 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 14:56:04.631765 23293 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 14:56:04.632673 23293 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 14:56:04.640599 23218 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 157c37291792449187ddf3ec01619734 [term 1 FOLLOWER]: Refusing update from remote peer ebd09ac4557a413dba8bcbfb01fcc73e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 14:56:04.641314 23291 consensus_queue.cc:1035] T 00000000000000000000000000000000 P ebd09ac4557a413dba8bcbfb01fcc73e [LEADER]: Connected to new peer: Peer: permanent_uuid: "157c37291792449187ddf3ec01619734" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37403 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 14:56:04.645641 23251 sys_catalog.cc:455] T 00000000000000000000000000000000 P 157c37291792449187ddf3ec01619734 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ebd09ac4557a413dba8bcbfb01fcc73e. Latest consensus state: current_term: 1 leader_uuid: "ebd09ac4557a413dba8bcbfb01fcc73e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebd09ac4557a413dba8bcbfb01fcc73e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45019 } } peers { permanent_uuid: "b3d14199395641b5a35166c3ab10ed38" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37771 } } peers { permanent_uuid: "157c37291792449187ddf3ec01619734" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37403 } } }
I20260927 14:56:04.645773 23251 sys_catalog.cc:458] T 00000000000000000000000000000000 P 157c37291792449187ddf3ec01619734 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 14:56:04.646076 23143 raft_consensus.cc:1261] T 00000000000000000000000000000000 P b3d14199395641b5a35166c3ab10ed38 [term 1 FOLLOWER]: Refusing update from remote peer ebd09ac4557a413dba8bcbfb01fcc73e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 14:56:04.646648 23291 consensus_queue.cc:1035] T 00000000000000000000000000000000 P ebd09ac4557a413dba8bcbfb01fcc73e [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3d14199395641b5a35166c3ab10ed38" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37771 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 14:56:04.650287 23265 sys_catalog.cc:455] T 00000000000000000000000000000000 P b3d14199395641b5a35166c3ab10ed38 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ebd09ac4557a413dba8bcbfb01fcc73e. Latest consensus state: current_term: 1 leader_uuid: "ebd09ac4557a413dba8bcbfb01fcc73e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebd09ac4557a413dba8bcbfb01fcc73e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45019 } } peers { permanent_uuid: "b3d14199395641b5a35166c3ab10ed38" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37771 } } peers { permanent_uuid: "157c37291792449187ddf3ec01619734" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37403 } } }
I20260927 14:56:04.650411 23265 sys_catalog.cc:458] T 00000000000000000000000000000000 P b3d14199395641b5a35166c3ab10ed38 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 14:56:04.651196 23249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:56:04.626301 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41088 (local address 127.0.98.60:37403)
0927 14:56:04.626411 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0927 14:56:04.626416 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 14:56:04.630140 (+ 3724us) server_negotiation.cc:407] Connection header received
0927 14:56:04.630179 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:56:04.630182 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:56:04.630234 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:56:04.630317 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:56:04.641167 (+ 10850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:56:04.642359 (+ 1192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:56:04.643485 (+ 1126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:56:04.643885 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:56:04.643968 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:56:04.646721 (+ 2753us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:56:04.646725 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:56:04.646728 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 14:56:04.646756 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:56:04.648535 (+ 1779us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:56:04.649085 (+ 550us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:56:04.649089 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:56:04.649091 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:56:04.649178 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:56:04.649485 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:56:04.649488 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:56:04.649490 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:56:04.649716 (+ 226us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:56:04.649834 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0927 14:56:04.650889 (+ 1055us) server_negotiation.cc:299] Negotiation successful
0927 14:56:04.651035 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260927 14:56:04.652825 23174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:56:04.626049 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53972 (local address 127.0.98.61:37771)
0927 14:56:04.626185 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0927 14:56:04.626189 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 14:56:04.629752 (+ 3563us) server_negotiation.cc:407] Connection header received
0927 14:56:04.629853 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:56:04.629856 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:56:04.629921 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:56:04.630002 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:56:04.637847 (+ 7845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:56:04.638643 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:56:04.639784 (+ 1141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:56:04.640132 (+ 348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:56:04.640221 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:56:04.642188 (+ 1967us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:56:04.642192 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:56:04.642195 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 14:56:04.642223 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:56:04.644373 (+ 2150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:56:04.650292 (+ 5919us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:56:04.650296 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:56:04.650298 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:56:04.650390 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:56:04.651638 (+ 1248us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:56:04.651642 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:56:04.651644 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:56:04.651870 (+ 226us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:56:04.652006 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0927 14:56:04.652554 (+ 548us) server_negotiation.cc:299] Negotiation successful
0927 14:56:04.652675 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260927 14:56:04.653779 23097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:56:04.626632 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54906 (local address 127.0.98.62:45019)
0927 14:56:04.627189 (+ 557us) server_negotiation.cc:206] Beginning negotiation
0927 14:56:04.627194 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 14:56:04.629483 (+ 2289us) server_negotiation.cc:407] Connection header received
0927 14:56:04.629582 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:56:04.629585 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:56:04.629639 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:56:04.629717 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:56:04.630750 (+ 1033us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:56:04.631465 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:56:04.632800 (+ 1335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:56:04.633188 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:56:04.633268 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:56:04.635115 (+ 1847us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:56:04.635119 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:56:04.635121 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 14:56:04.635147 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:56:04.636830 (+ 1683us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:56:04.651770 (+ 14940us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:56:04.651774 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:56:04.651776 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:56:04.651876 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:56:04.652231 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:56:04.652235 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:56:04.652237 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:56:04.652485 (+ 248us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:56:04.652603 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0927 14:56:04.653508 (+ 905us) server_negotiation.cc:299] Negotiation successful
0927 14:56:04.653630 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":484}
I20260927 14:56:04.657541 23291 sys_catalog.cc:455] T 00000000000000000000000000000000 P ebd09ac4557a413dba8bcbfb01fcc73e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "ebd09ac4557a413dba8bcbfb01fcc73e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebd09ac4557a413dba8bcbfb01fcc73e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45019 } } peers { permanent_uuid: "b3d14199395641b5a35166c3ab10ed38" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37771 } } peers { permanent_uuid: "157c37291792449187ddf3ec01619734" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37403 } } }
I20260927 14:56:04.657661 23291 sys_catalog.cc:458] T 00000000000000000000000000000000 P ebd09ac4557a413dba8bcbfb01fcc73e [sys.catalog]: This master's current role is: LEADER
I20260927 14:56:04.657691 23302 mvcc.cc:205] Tried to move back new op lower bound from 7333973871160143872 to 7333973871088783360. Current Snapshot: MvccSnapshot[applied={T|T < 7333973871160143872}]
I20260927 14:56:04.657779 23291 sys_catalog.cc:455] T 00000000000000000000000000000000 P ebd09ac4557a413dba8bcbfb01fcc73e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "ebd09ac4557a413dba8bcbfb01fcc73e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebd09ac4557a413dba8bcbfb01fcc73e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45019 } } peers { permanent_uuid: "b3d14199395641b5a35166c3ab10ed38" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37771 } } peers { permanent_uuid: "157c37291792449187ddf3ec01619734" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37403 } } }
I20260927 14:56:04.657836 23291 sys_catalog.cc:458] T 00000000000000000000000000000000 P ebd09ac4557a413dba8bcbfb01fcc73e [sys.catalog]: This master's current role is: LEADER
I20260927 14:56:04.657874 23265 sys_catalog.cc:455] T 00000000000000000000000000000000 P b3d14199395641b5a35166c3ab10ed38 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "ebd09ac4557a413dba8bcbfb01fcc73e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebd09ac4557a413dba8bcbfb01fcc73e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45019 } } peers { permanent_uuid: "b3d14199395641b5a35166c3ab10ed38" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37771 } } peers { permanent_uuid: "157c37291792449187ddf3ec01619734" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37403 } } }
I20260927 14:56:04.657989 23265 sys_catalog.cc:458] T 00000000000000000000000000000000 P b3d14199395641b5a35166c3ab10ed38 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 14:56:04.658095 23305 mvcc.cc:205] Tried to move back new op lower bound from 7333973871160143872 to 7333973871088783360. Current Snapshot: MvccSnapshot[applied={T|T < 7333973871160143872}]
I20260927 14:56:04.658829 23251 sys_catalog.cc:455] T 00000000000000000000000000000000 P 157c37291792449187ddf3ec01619734 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "ebd09ac4557a413dba8bcbfb01fcc73e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebd09ac4557a413dba8bcbfb01fcc73e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45019 } } peers { permanent_uuid: "b3d14199395641b5a35166c3ab10ed38" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37771 } } peers { permanent_uuid: "157c37291792449187ddf3ec01619734" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37403 } } }
I20260927 14:56:04.658946 23251 sys_catalog.cc:458] T 00000000000000000000000000000000 P 157c37291792449187ddf3ec01619734 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 14:56:04.660561 23293 catalog_manager.cc:1347] Generated new cluster ID: 693c05b709844fb589166d36b0548932
I20260927 14:56:04.660619 23293 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 14:56:04.691915 23293 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 14:56:04.693292 23293 catalog_manager.cc:1504] Loading token signing keys...
I20260927 14:56:04.704068 23293 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ebd09ac4557a413dba8bcbfb01fcc73e: Generated new TSK 0
I20260927 14:56:04.704906 23293 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260927 14:56:04.732663 23097 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 14:56:04.729516 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54916 (local address 127.0.98.62:45019)
0927 14:56:04.729645 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0927 14:56:04.729650 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 14:56:04.731555 (+ 1905us) server_negotiation.cc:407] Connection header received
0927 14:56:04.731763 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:56:04.731767 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:56:04.731831 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:56:04.731886 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:56:04.732520 (+ 634us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54916: BlockingRecv error: recv got EOF from 127.0.0.1:54916 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
W20260927 14:56:04.734056 23174 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 14:56:04.730391 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53986 (local address 127.0.98.61:37771)
0927 14:56:04.730502 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0927 14:56:04.730506 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 14:56:04.733177 (+ 2671us) server_negotiation.cc:407] Connection header received
0927 14:56:04.733344 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:56:04.733347 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:56:04.733397 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:56:04.733447 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:56:04.733937 (+ 490us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53986: BlockingRecv error: recv got EOF from 127.0.0.1:53986 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
W20260927 14:56:04.735517 23249 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 14:56:04.730995 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41098 (local address 127.0.98.60:37403)
0927 14:56:04.731105 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0927 14:56:04.731109 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 14:56:04.734430 (+ 3321us) server_negotiation.cc:407] Connection header received
0927 14:56:04.734782 (+ 352us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:56:04.734785 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:56:04.734849 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:56:04.734905 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:56:04.735406 (+ 501us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41098: BlockingRecv error: recv got EOF from 127.0.0.1:41098 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
W20260927 14:56:04.746305 23174 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 14:56:04.741427 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53990 (local address 127.0.98.61:37771)
0927 14:56:04.741539 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0927 14:56:04.741543 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 14:56:04.745339 (+ 3796us) server_negotiation.cc:407] Connection header received
0927 14:56:04.745555 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:56:04.745559 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:56:04.745635 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:56:04.745687 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:56:04.746201 (+ 514us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53990: BlockingRecv error: recv got EOF from 127.0.0.1:53990 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
W20260927 14:56:04.751206 23174 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 14:56:04.749827 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54002 (local address 127.0.98.61:37771)
0927 14:56:04.749928 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0927 14:56:04.749932 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 14:56:04.750377 (+ 445us) server_negotiation.cc:407] Connection header received
0927 14:56:04.750540 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:56:04.750543 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:56:04.750589 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:56:04.750635 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:56:04.751115 (+ 480us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54002: BlockingRecv error: recv got EOF from 127.0.0.1:54002 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
W20260927 14:56:04.756608 23174 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 14:56:04.755171 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54012 (local address 127.0.98.61:37771)
0927 14:56:04.755263 (+ 92us) server_negotiation.cc:206] Beginning negotiation
0927 14:56:04.755267 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 14:56:04.755790 (+ 523us) server_negotiation.cc:407] Connection header received
0927 14:56:04.755974 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:56:04.755977 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:56:04.756020 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:56:04.756067 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:56:04.756514 (+ 447us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54012: BlockingRecv error: recv got EOF from 127.0.0.1:54012 (error 108)
Metrics: {"server-negotiator.queue_time_us":36}
W20260927 14:56:04.764221 23174 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 14:56:04.762676 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54022 (local address 127.0.98.61:37771)
0927 14:56:04.762808 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0927 14:56:04.762812 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 14:56:04.763366 (+ 554us) server_negotiation.cc:407] Connection header received
0927 14:56:04.763542 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:56:04.763545 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:56:04.763590 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:56:04.763638 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:56:04.764122 (+ 484us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54022: BlockingRecv error: recv got EOF from 127.0.0.1:54022 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
W20260927 14:56:04.776100 23174 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 14:56:04.774551 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54034 (local address 127.0.98.61:37771)
0927 14:56:04.774662 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0927 14:56:04.774667 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 14:56:04.775174 (+ 507us) server_negotiation.cc:407] Connection header received
0927 14:56:04.775368 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:56:04.775371 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:56:04.775421 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:56:04.775471 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:56:04.775991 (+ 520us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54034: BlockingRecv error: recv got EOF from 127.0.0.1:54034 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
W20260927 14:56:04.796638 23174 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 14:56:04.794784 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54044 (local address 127.0.98.61:37771)
0927 14:56:04.794928 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0927 14:56:04.794932 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 14:56:04.795538 (+ 606us) server_negotiation.cc:407] Connection header received
0927 14:56:04.795747 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:56:04.795750 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:56:04.795803 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:56:04.795873 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:56:04.796500 (+ 627us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54044: BlockingRecv error: recv got EOF from 127.0.0.1:54044 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
W20260927 14:56:04.832755 23174 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 14:56:04.831442 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54056 (local address 127.0.98.61:37771)
0927 14:56:04.831565 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0927 14:56:04.831569 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 14:56:04.831844 (+ 275us) server_negotiation.cc:407] Connection header received
0927 14:56:04.831991 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:56:04.831995 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:56:04.832058 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:56:04.832153 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:56:04.832632 (+ 479us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54056: BlockingRecv error: recv got EOF from 127.0.0.1:54056 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
W20260927 14:56:04.899866 23174 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 14:56:04.898627 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54064 (local address 127.0.98.61:37771)
0927 14:56:04.898807 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0927 14:56:04.898812 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 14:56:04.898955 (+ 143us) server_negotiation.cc:407] Connection header received
0927 14:56:04.899135 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:56:04.899138 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:56:04.899194 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:56:04.899280 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:56:04.899740 (+ 460us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54064: BlockingRecv error: recv got EOF from 127.0.0.1:54064 (error 108)
Metrics: {"server-negotiator.queue_time_us":82}
W20260927 14:56:05.031121 23174 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 14:56:05.029927 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54076 (local address 127.0.98.61:37771)
0927 14:56:05.030094 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0927 14:56:05.030099 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 14:56:05.030243 (+ 144us) server_negotiation.cc:407] Connection header received
0927 14:56:05.030410 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:56:05.030413 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:56:05.030466 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:56:05.030564 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:56:05.031001 (+ 437us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54076: BlockingRecv error: recv got EOF from 127.0.0.1:54076 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
W20260927 14:56:05.290228 23174 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 14:56:05.289127 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54080 (local address 127.0.98.61:37771)
0927 14:56:05.289259 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0927 14:56:05.289265 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 14:56:05.289351 (+ 86us) server_negotiation.cc:407] Connection header received
0927 14:56:05.289519 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:56:05.289522 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:56:05.289579 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:56:05.289671 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:56:05.290109 (+ 438us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54080: BlockingRecv error: recv got EOF from 127.0.0.1:54080 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
I20260927 14:56:05.603125 23263 catalog_manager.cc:1259] Loaded cluster ID: 693c05b709844fb589166d36b0548932
I20260927 14:56:05.603211 23263 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 157c37291792449187ddf3ec01619734: loading cluster ID for follower catalog manager: success
I20260927 14:56:05.604867 23263 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 157c37291792449187ddf3ec01619734: acquiring CA information for follower catalog manager: success
I20260927 14:56:05.606125 23263 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 157c37291792449187ddf3ec01619734: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260927 14:56:05.625561 23277 catalog_manager.cc:1259] Loaded cluster ID: 693c05b709844fb589166d36b0548932
I20260927 14:56:05.625629 23277 catalog_manager.cc:1552] T 00000000000000000000000000000000 P b3d14199395641b5a35166c3ab10ed38: loading cluster ID for follower catalog manager: success
I20260927 14:56:05.627136 23277 catalog_manager.cc:1574] T 00000000000000000000000000000000 P b3d14199395641b5a35166c3ab10ed38: acquiring CA information for follower catalog manager: success
I20260927 14:56:05.628185 23277 catalog_manager.cc:1602] T 00000000000000000000000000000000 P b3d14199395641b5a35166c3ab10ed38: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260927 14:56:05.808405 23375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:56:05.804278 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54082 (local address 127.0.98.61:37771)
0927 14:56:05.804566 (+ 288us) server_negotiation.cc:206] Beginning negotiation
0927 14:56:05.804574 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 14:56:05.804591 (+ 17us) server_negotiation.cc:407] Connection header received
0927 14:56:05.804755 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:56:05.804758 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:56:05.804888 (+ 130us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:56:05.805003 (+ 115us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 14:56:05.805291 (+ 288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:56:05.805854 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:56:05.806774 (+ 920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:56:05.807001 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:56:05.807126 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:56:05.807423 (+ 297us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 14:56:05.807856 (+ 433us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 14:56:05.807989 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0927 14:56:05.808124 (+ 135us) server_negotiation.cc:299] Negotiation successful
0927 14:56:05.808216 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":179,"thread_start_us":84,"threads_started":1}
I20260927 14:56:05.814939 23381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:56:05.811439 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41106 (local address 127.0.98.60:37403)
0927 14:56:05.811642 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0927 14:56:05.811646 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 14:56:05.811659 (+ 13us) server_negotiation.cc:407] Connection header received
0927 14:56:05.811735 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:56:05.811737 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:56:05.811847 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:56:05.811948 (+ 101us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 14:56:05.812121 (+ 173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:56:05.812610 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:56:05.813516 (+ 906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:56:05.813732 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:56:05.813812 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:56:05.814122 (+ 310us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 14:56:05.814573 (+ 451us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 14:56:05.814671 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0927 14:56:05.814722 (+ 51us) server_negotiation.cc:299] Negotiation successful
0927 14:56:05.814768 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":143,"thread_start_us":78,"threads_started":1}
I20260927 14:56:05.821987 23375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:56:05.818621 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54098 (local address 127.0.98.61:37771)
0927 14:56:05.818756 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0927 14:56:05.818761 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 14:56:05.818773 (+ 12us) server_negotiation.cc:407] Connection header received
0927 14:56:05.818817 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:56:05.818819 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:56:05.818864 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:56:05.818934 (+ 70us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 14:56:05.819255 (+ 321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:56:05.819901 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:56:05.820753 (+ 852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:56:05.821112 (+ 359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:56:05.821200 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:56:05.821520 (+ 320us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 14:56:05.821598 (+ 78us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 14:56:05.821667 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0927 14:56:05.821767 (+ 100us) server_negotiation.cc:299] Negotiation successful
0927 14:56:05.821821 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260927 14:56:05.822327 23381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:56:05.818473 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41110 (local address 127.0.98.60:37403)
0927 14:56:05.818593 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0927 14:56:05.818597 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 14:56:05.818886 (+ 289us) server_negotiation.cc:407] Connection header received
0927 14:56:05.818983 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:56:05.818986 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:56:05.819030 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:56:05.819104 (+ 74us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 14:56:05.819337 (+ 233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:56:05.819985 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:56:05.821123 (+ 1138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:56:05.821429 (+ 306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:56:05.821501 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:56:05.821845 (+ 344us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 14:56:05.821923 (+ 78us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 14:56:05.821968 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0927 14:56:05.822157 (+ 189us) server_negotiation.cc:299] Negotiation successful
0927 14:56:05.822205 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260927 14:56:05.822638 23387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:56:05.817856 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54926 (local address 127.0.98.62:45019)
0927 14:56:05.818140 (+ 284us) server_negotiation.cc:206] Beginning negotiation
0927 14:56:05.818147 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 14:56:05.818161 (+ 14us) server_negotiation.cc:407] Connection header received
0927 14:56:05.818242 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:56:05.818244 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:56:05.818415 (+ 171us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:56:05.818541 (+ 126us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 14:56:05.818762 (+ 221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:56:05.819552 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:56:05.820701 (+ 1149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:56:05.820956 (+ 255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:56:05.821101 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:56:05.821367 (+ 266us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 14:56:05.821691 (+ 324us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 14:56:05.821766 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0927 14:56:05.822418 (+ 652us) server_negotiation.cc:299] Negotiation successful
0927 14:56:05.822474 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":194,"thread_start_us":93,"threads_started":1}
I20260927 14:56:05.825548 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6936m5/build/debug/bin/kudu with pid 23026
I20260927 14:56:05.831745 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6936m5/build/debug/bin/kudu with pid 23101
I20260927 14:56:05.837713 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6936m5/build/debug/bin/kudu with pid 23175
2026-09-27T14:56:05Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (3042 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (6093 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (281228 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST